2026-02-17T04:45:26.657 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-17T04:45:26.672 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-17T04:45:26.750 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/skanta-2026-02-17_03:53:03-rados-wip-bharath18-testing-2026-02-13-0856-distro-default-trial/50912 branch: wip-bharath18-testing-2026-02-13-0856 description: rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/many msgr/async-v2only objectstore/{bluestore/{alloc$/{stupid} base mem$/{normal-1} onode-segment$/{1M} write$/{v2/{compr$/{no$/{no}} v2}}}} rados supported-random-distro$/{centos_latest}} email: skanta@redhat.com first_in_suite: false flavor: default job_id: '50912' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: skanta-2026-02-17_03:53:03-rados-wip-bharath18-testing-2026-02-13-0856-distro-default-trial no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath18-testing-2026-02-13-0856 ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon election default strategy: 1 mon mgr beacon grace: 90 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 30 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore onode segment size: 1024K bluestore write v2: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) - \(MON_NETSPLIT\) sha1: 325850157d309f65985eea9ec9bb9f1d56726fee ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 325850157d309f65985eea9ec9bb9f1d56726fee thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath18-testing-2026-02-13-0856 sha1: 325850157d309f65985eea9ec9bb9f1d56726fee owner: scheduled_skanta@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 7109 sha1: 325850157d309f65985eea9ec9bb9f1d56726fee sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath18-testing-2026-02-13-0856 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_325850157d309f65985eea9ec9bb9f1d56726fee/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 325850157d309f65985eea9ec9bb9f1d56726fee targets: trial100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOUTuXxdAZDZd8keUQ+KsOPAQTdKnJRgpaS9f2A2d8TgRPHE3RLiXCB/DgWNkwWwODPFxfNIaty3XpzTER+SCE= tasks: - install: null - ceph: conf: osd: osd max pg log entries: 5 osd min pg log entries: 5 create_rbd_pool: false log-ignorelist: - Error - overall HEALTH_ - \(OBJECT_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_BACKFILLFULL\) - \(POOL_NEARFULL\) - \(SLOW_OPS\) - \(TOO_FEW_PGS\) - Monitor daemon marked osd\.[[:digit:]]+ down, but it is still running - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force - backfill_toofull: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-17_03:53:03 tube: trial user: skanta verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3682 2026-02-17T04:45:26.750 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_325850157d309f65985eea9ec9bb9f1d56726fee/qa; will attempt to use it 2026-02-17T04:45:26.750 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_325850157d309f65985eea9ec9bb9f1d56726fee/qa/tasks 2026-02-17T04:45:26.751 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-17T04:45:26.755 INFO:teuthology.task.internal:Checking packages... 2026-02-17T04:45:26.755 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '325850157d309f65985eea9ec9bb9f1d56726fee' 2026-02-17T04:45:26.755 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-17T04:45:26.755 INFO:teuthology.packaging:ref: None 2026-02-17T04:45:26.755 INFO:teuthology.packaging:tag: None 2026-02-17T04:45:26.755 INFO:teuthology.packaging:branch: wip-bharath18-testing-2026-02-13-0856 2026-02-17T04:45:26.755 INFO:teuthology.packaging:sha1: 325850157d309f65985eea9ec9bb9f1d56726fee 2026-02-17T04:45:26.755 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath18-testing-2026-02-13-0856 2026-02-17T04:45:26.878 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5271.g32585015 2026-02-17T04:45:26.880 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-17T04:45:26.902 INFO:teuthology.task.internal:no buildpackages task found 2026-02-17T04:45:26.902 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-17T04:45:26.914 INFO:teuthology.task.internal:Saving configuration 2026-02-17T04:45:26.924 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-17T04:45:26.936 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-17T04:45:27.001 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial100.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/skanta-2026-02-17_03:53:03-rados-wip-bharath18-testing-2026-02-13-0856-distro-default-trial/50912', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-17 04:41:57.802962', 'locked_by': 'scheduled_skanta@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOUTuXxdAZDZd8keUQ+KsOPAQTdKnJRgpaS9f2A2d8TgRPHE3RLiXCB/DgWNkwWwODPFxfNIaty3XpzTER+SCE='} 2026-02-17T04:45:27.001 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-17T04:45:27.016 INFO:teuthology.task.internal:roles: ubuntu@trial100.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2026-02-17T04:45:27.016 INFO:teuthology.run_tasks:Running task console_log... 2026-02-17T04:45:27.091 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x703d5b0fafc0>, signals=[15]) 2026-02-17T04:45:27.091 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-17T04:45:27.108 INFO:teuthology.task.internal:Opening connections... 2026-02-17T04:45:27.108 DEBUG:teuthology.task.internal:connecting to ubuntu@trial100.front.sepia.ceph.com 2026-02-17T04:45:27.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-17T04:45:27.183 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-17T04:45:27.194 DEBUG:teuthology.orchestra.run.trial100:> uname -m 2026-02-17T04:45:27.264 INFO:teuthology.orchestra.run.trial100.stdout:x86_64 2026-02-17T04:45:27.264 DEBUG:teuthology.orchestra.run.trial100:> cat /etc/os-release 2026-02-17T04:45:27.320 INFO:teuthology.orchestra.run.trial100.stdout:NAME="CentOS Stream" 2026-02-17T04:45:27.320 INFO:teuthology.orchestra.run.trial100.stdout:VERSION="9" 2026-02-17T04:45:27.320 INFO:teuthology.orchestra.run.trial100.stdout:ID="centos" 2026-02-17T04:45:27.320 INFO:teuthology.orchestra.run.trial100.stdout:ID_LIKE="rhel fedora" 2026-02-17T04:45:27.320 INFO:teuthology.orchestra.run.trial100.stdout:VERSION_ID="9" 2026-02-17T04:45:27.320 INFO:teuthology.orchestra.run.trial100.stdout:PLATFORM_ID="platform:el9" 2026-02-17T04:45:27.320 INFO:teuthology.orchestra.run.trial100.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-17T04:45:27.320 INFO:teuthology.orchestra.run.trial100.stdout:ANSI_COLOR="0;31" 2026-02-17T04:45:27.321 INFO:teuthology.orchestra.run.trial100.stdout:LOGO="fedora-logo-icon" 2026-02-17T04:45:27.321 INFO:teuthology.orchestra.run.trial100.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-17T04:45:27.321 INFO:teuthology.orchestra.run.trial100.stdout:HOME_URL="https://centos.org/" 2026-02-17T04:45:27.321 INFO:teuthology.orchestra.run.trial100.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-17T04:45:27.321 INFO:teuthology.orchestra.run.trial100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-17T04:45:27.321 INFO:teuthology.orchestra.run.trial100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-17T04:45:27.321 INFO:teuthology.lock.ops:Updating trial100.front.sepia.ceph.com on lock server 2026-02-17T04:45:27.405 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-17T04:45:27.417 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-17T04:45:27.428 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-17T04:45:27.428 DEBUG:teuthology.orchestra.run.trial100:> test '!' -e /home/ubuntu/cephtest 2026-02-17T04:45:27.444 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-17T04:45:27.454 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-17T04:45:27.455 DEBUG:teuthology.orchestra.run.trial100:> test -z $(ls -A /var/lib/ceph) 2026-02-17T04:45:27.500 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-17T04:45:27.595 INFO:teuthology.run_tasks:Running task kernel... 2026-02-17T04:45:27.617 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-17T04:45:27.617 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-17T04:45:27.617 DEBUG:teuthology.orchestra.run.trial100:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-17T04:45:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-17T04:45:27.632 DEBUG:teuthology.orchestra.run.trial100:> uname -r 2026-02-17T04:45:27.688 INFO:teuthology.orchestra.run.trial100.stdout:5.14.0-645.el9.x86_64 2026-02-17T04:45:27.688 INFO:teuthology.task.kernel:Running kernel on trial100: 5.14.0-645.el9.x86_64 2026-02-17T04:45:27.688 DEBUG:teuthology.orchestra.run.trial100:> sudo yum install -y kernel 2026-02-17T04:45:28.780 INFO:teuthology.orchestra.run.trial100.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-17T04:45:57.030 INFO:teuthology.orchestra.run.trial100.stdout:CentOS Stream 9 - AppStream 991 kB/s | 27 MB 00:27 2026-02-17T04:46:08.580 INFO:teuthology.orchestra.run.trial100.stdout:CentOS Stream 9 - CRB 872 kB/s | 7.8 MB 00:09 2026-02-17T04:46:09.435 INFO:teuthology.orchestra.run.trial100.stdout:CentOS Stream 9 - Extras packages 153 kB/s | 20 kB 00:00 2026-02-17T04:46:27.159 INFO:teuthology.orchestra.run.trial100.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 20 MB 00:17 2026-02-17T04:46:30.359 INFO:teuthology.orchestra.run.trial100.stdout:lab-extras 1.3 MB/s | 50 kB 00:00 2026-02-17T04:46:31.386 INFO:teuthology.orchestra.run.trial100.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-17T04:46:31.406 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout: Package Architecture Version Repository Size 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout:Installing: 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout:Installing dependencies: 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout: kernel-core x86_64 5.14.0-681.el9 baseos 17 M 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout: kernel-modules x86_64 5.14.0-681.el9 baseos 40 M 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout: kernel-modules-core x86_64 5.14.0-681.el9 baseos 31 M 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout:Transaction Summary 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout:Install 4 Packages 2026-02-17T04:46:31.407 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:46:31.408 INFO:teuthology.orchestra.run.trial100.stdout:Total download size: 88 M 2026-02-17T04:46:31.408 INFO:teuthology.orchestra.run.trial100.stdout:Installed size: 135 M 2026-02-17T04:46:31.408 INFO:teuthology.orchestra.run.trial100.stdout:Downloading Packages: 2026-02-17T04:46:33.113 INFO:teuthology.orchestra.run.trial100.stdout:(1/4): kernel-5.14.0-681.el9.x86_64.rpm 495 kB/s | 803 kB 00:01 2026-02-17T04:46:53.349 INFO:teuthology.orchestra.run.trial100.stdout:(2/4): kernel-core-5.14.0-681.el9.x86_64.rpm 800 kB/s | 17 MB 00:21 2026-02-17T04:47:10.110 INFO:teuthology.orchestra.run.trial100.stdout:(3/4): kernel-modules-core-5.14.0-681.el9.x86_6 854 kB/s | 31 MB 00:36 2026-02-17T04:47:18.743 INFO:teuthology.orchestra.run.trial100.stdout:(4/4): kernel-modules-5.14.0-681.el9.x86_64.rpm 858 kB/s | 40 MB 00:47 2026-02-17T04:47:18.745 INFO:teuthology.orchestra.run.trial100.stdout:-------------------------------------------------------------------------------- 2026-02-17T04:47:18.745 INFO:teuthology.orchestra.run.trial100.stdout:Total 1.9 MB/s | 88 MB 00:47 2026-02-17T04:47:18.935 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction check 2026-02-17T04:47:19.132 INFO:teuthology.orchestra.run.trial100.stdout:Transaction check succeeded. 2026-02-17T04:47:19.133 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction test 2026-02-17T04:47:19.478 INFO:teuthology.orchestra.run.trial100.stdout:Transaction test succeeded. 2026-02-17T04:47:19.478 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction 2026-02-17T04:47:20.567 INFO:teuthology.orchestra.run.trial100.stdout: Preparing : 1/1 2026-02-17T04:47:20.852 INFO:teuthology.orchestra.run.trial100.stdout: Installing : kernel-modules-core-5.14.0-681.el9.x86_64 1/4 2026-02-17T04:47:20.872 INFO:teuthology.orchestra.run.trial100.stdout: Installing : kernel-core-5.14.0-681.el9.x86_64 2/4 2026-02-17T04:47:22.650 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: kernel-core-5.14.0-681.el9.x86_64 2/4 2026-02-17T04:47:22.679 INFO:teuthology.orchestra.run.trial100.stdout: Installing : kernel-modules-5.14.0-681.el9.x86_64 3/4 2026-02-17T04:47:25.765 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: kernel-modules-5.14.0-681.el9.x86_64 3/4 2026-02-17T04:47:25.771 INFO:teuthology.orchestra.run.trial100.stdout: Installing : kernel-5.14.0-681.el9.x86_64 4/4 2026-02-17T04:47:28.806 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: kernel-modules-core-5.14.0-681.el9.x86_64 4/4 2026-02-17T04:47:59.178 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: kernel-core-5.14.0-681.el9.x86_64 4/4 2026-02-17T04:47:59.198 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: kernel-modules-5.14.0-681.el9.x86_64 4/4 2026-02-17T04:47:59.261 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 4/4 2026-02-17T04:47:59.261 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/4 2026-02-17T04:47:59.261 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : kernel-core-5.14.0-681.el9.x86_64 2/4 2026-02-17T04:47:59.262 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : kernel-modules-5.14.0-681.el9.x86_64 3/4 2026-02-17T04:47:59.313 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : kernel-modules-core-5.14.0-681.el9.x86_64 4/4 2026-02-17T04:47:59.314 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:47:59.314 INFO:teuthology.orchestra.run.trial100.stdout:Installed: 2026-02-17T04:47:59.314 INFO:teuthology.orchestra.run.trial100.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-17T04:47:59.314 INFO:teuthology.orchestra.run.trial100.stdout: kernel-core-5.14.0-681.el9.x86_64 2026-02-17T04:47:59.314 INFO:teuthology.orchestra.run.trial100.stdout: kernel-modules-5.14.0-681.el9.x86_64 2026-02-17T04:47:59.314 INFO:teuthology.orchestra.run.trial100.stdout: kernel-modules-core-5.14.0-681.el9.x86_64 2026-02-17T04:47:59.314 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:47:59.314 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:47:59.372 DEBUG:teuthology.orchestra.run.trial100:> rpm -q kernel | sort -rV | head -n 1 2026-02-17T04:47:59.398 INFO:teuthology.orchestra.run.trial100.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-17T04:47:59.399 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-17T04:47:59.399 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-645.el9.x86_64 Expected: 5.14.0-681.el9.x86_64 2026-02-17T04:47:59.400 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-17T04:47:59.400 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-17T04:47:59.400 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-17T04:47:59.400 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial100.front.sepia.ceph.com, path=None, version=distro) 2026-02-17T04:47:59.400 DEBUG:teuthology.orchestra.run.trial100:> rpm -q kernel | sort -rV | head -n 1 2026-02-17T04:47:59.462 INFO:teuthology.orchestra.run.trial100.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-17T04:47:59.463 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-17T04:47:59.464 DEBUG:teuthology.orchestra.run.trial100:> sudo rpm -qi grub2-tools 2026-02-17T04:47:59.531 INFO:teuthology.orchestra.run.trial100.stdout:Name : grub2-tools 2026-02-17T04:47:59.531 INFO:teuthology.orchestra.run.trial100.stdout:Epoch : 1 2026-02-17T04:47:59.531 INFO:teuthology.orchestra.run.trial100.stdout:Version : 2.06 2026-02-17T04:47:59.531 INFO:teuthology.orchestra.run.trial100.stdout:Release : 118.el9 2026-02-17T04:47:59.531 INFO:teuthology.orchestra.run.trial100.stdout:Architecture: x86_64 2026-02-17T04:47:59.531 INFO:teuthology.orchestra.run.trial100.stdout:Install Date: Thu 04 Dec 2025 11:54:21 PM UTC 2026-02-17T04:47:59.531 INFO:teuthology.orchestra.run.trial100.stdout:Group : Unspecified 2026-02-17T04:47:59.531 INFO:teuthology.orchestra.run.trial100.stdout:Size : 8393909 2026-02-17T04:47:59.531 INFO:teuthology.orchestra.run.trial100.stdout:License : GPLv3+ 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:Signature : RSA/SHA256, Wed 05 Nov 2025 09:13:15 AM UTC, Key ID 05b555b38483c65d 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:Source RPM : grub2-2.06-118.el9.src.rpm 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:Build Date : Tue 04 Nov 2025 03:44:14 PM UTC 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:Packager : builder@centos.org 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:Vendor : CentOS 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:Summary : Support tools for GRUB. 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:Description : 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:hardware devices. 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:47:59.532 INFO:teuthology.orchestra.run.trial100.stdout:This subpackage provides tools for support of all platforms. 2026-02-17T04:47:59.533 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-17T04:47:59.533 INFO:teuthology.task.kernel:Updating grub on trial100 to boot 5.14.0-681.el9.x86_64 2026-02-17T04:47:59.533 DEBUG:teuthology.orchestra.run.trial100:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-17T04:47:59.597 INFO:teuthology.orchestra.run.trial100.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-17T04:47:59.598 DEBUG:teuthology.orchestra.run.trial100:> sudo command -v grubby && echo yes || echo no 2026-02-17T04:47:59.662 INFO:teuthology.orchestra.run.trial100.stdout:/sbin/grubby 2026-02-17T04:47:59.663 INFO:teuthology.orchestra.run.trial100.stdout:yes 2026-02-17T04:47:59.663 DEBUG:teuthology.orchestra.run.trial100:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-681.el9.x86_64" | head -n 1 2026-02-17T04:47:59.729 INFO:teuthology.orchestra.run.trial100.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-17T04:47:59.729 DEBUG:teuthology.orchestra.run.trial100:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64 2026-02-17T04:47:59.801 DEBUG:teuthology.orchestra.run.trial100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-17T04:48:00.012 INFO:teuthology.orchestra.run.trial100.stderr:Generating grub configuration file ... 2026-02-17T04:48:00.076 INFO:teuthology.orchestra.run.trial100.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-17T04:48:00.082 INFO:teuthology.orchestra.run.trial100.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-17T04:48:00.333 INFO:teuthology.orchestra.run.trial100.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-17T04:48:00.340 INFO:teuthology.orchestra.run.trial100.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-17T04:48:00.371 INFO:teuthology.orchestra.run.trial100.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-17T04:48:00.378 INFO:teuthology.orchestra.run.trial100.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-17T04:48:00.401 INFO:teuthology.orchestra.run.trial100.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-17T04:48:00.407 INFO:teuthology.orchestra.run.trial100.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-17T04:48:00.491 INFO:teuthology.orchestra.run.trial100.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-17T04:48:00.499 INFO:teuthology.orchestra.run.trial100.stderr:done 2026-02-17T04:48:00.501 DEBUG:teuthology.orchestra.run.trial100:> test -d /sys/firmware/efi 2026-02-17T04:48:00.516 DEBUG:teuthology.orchestra.run.trial100:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-17T04:48:00.581 INFO:teuthology.orchestra.run.trial100.stdout:/boot/efi/EFI/BOOT 2026-02-17T04:48:00.581 INFO:teuthology.orchestra.run.trial100.stdout:/boot/efi/EFI/centos 2026-02-17T04:48:00.582 DEBUG:teuthology.orchestra.run.trial100:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-17T04:48:00.649 INFO:teuthology.orchestra.run.trial100.stdout:no 2026-02-17T04:48:00.650 DEBUG:teuthology.orchestra.run.trial100:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-17T04:48:00.713 INFO:teuthology.orchestra.run.trial100.stdout:yes 2026-02-17T04:48:00.714 DEBUG:teuthology.orchestra.run.trial100:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-17T04:48:00.777 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial100 2026-02-17T04:48:00.778 DEBUG:teuthology.orchestra.run.trial100:> sudo bash -c ' 2026-02-17T04:48:00.778 DEBUG:teuthology.orchestra.run.trial100:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-17T04:48:00.778 DEBUG:teuthology.orchestra.run.trial100:> ' 2026-02-17T04:48:30.820 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-17T04:48:30.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial100.front.sepia.ceph.com' 2026-02-17T04:48:30.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-17T04:48:45.121 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.100 2026-02-17T04:48:54.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial100.front.sepia.ceph.com' 2026-02-17T04:48:54.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-17T04:49:30.112 DEBUG:teuthology.orchestra.run.trial100:> true 2026-02-17T04:49:30.260 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial100.front.sepia.ceph.com' 2026-02-17T04:49:30.260 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-681.el9.x86_64"... 2026-02-17T04:49:30.260 DEBUG:teuthology.orchestra.run.trial100:> uname -r 2026-02-17T04:49:30.315 INFO:teuthology.orchestra.run.trial100.stdout:5.14.0-681.el9.x86_64 2026-02-17T04:49:30.316 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-681.el9.x86_64 vs 5.14.0-681.el9.x86_64 2026-02-17T04:49:30.316 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-17T04:49:30.316 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-17T04:49:31.317 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-17T04:49:31.317 DEBUG:teuthology.orchestra.run.trial100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-17T04:49:31.346 INFO:teuthology.orchestra.run.trial100.stdout:ttyS1 2026-02-17T04:49:31.368 DEBUG:teuthology.parallel:result is None 2026-02-17T04:49:31.368 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-17T04:49:31.379 INFO:teuthology.task.internal:Creating test directory... 2026-02-17T04:49:31.379 DEBUG:teuthology.orchestra.run.trial100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-17T04:49:31.423 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-17T04:49:31.435 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-17T04:49:31.451 INFO:teuthology.task.internal:Creating archive directory... 2026-02-17T04:49:31.451 DEBUG:teuthology.orchestra.run.trial100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-17T04:49:31.487 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-17T04:49:31.499 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-17T04:49:31.499 DEBUG:teuthology.orchestra.run.trial100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-17T04:49:31.546 INFO:teuthology.orchestra.run.trial100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-17T04:49:31.554 INFO:teuthology.orchestra.run.trial100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-17T04:49:31.556 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-17T04:49:31.568 INFO:teuthology.task.internal:Configuring sudo... 2026-02-17T04:49:31.568 DEBUG:teuthology.orchestra.run.trial100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-17T04:49:31.620 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-17T04:49:31.638 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-17T04:49:31.638 DEBUG:teuthology.orchestra.run.trial100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-17T04:49:31.677 DEBUG:teuthology.orchestra.run.trial100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-17T04:49:31.839 DEBUG:teuthology.orchestra.run.trial100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-17T04:49:31.865 DEBUG:teuthology.orchestra.run.trial100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-17T04:49:32.005 DEBUG:teuthology.orchestra.run.trial100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-17T04:49:32.030 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:49:32.030 DEBUG:teuthology.orchestra.run.trial100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-17T04:49:32.092 DEBUG:teuthology.orchestra.run.trial100:> sudo service rsyslog restart 2026-02-17T04:49:32.161 INFO:teuthology.orchestra.run.trial100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-17T04:49:32.402 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-17T04:49:32.417 INFO:teuthology.task.internal:Starting timer... 2026-02-17T04:49:32.417 INFO:teuthology.run_tasks:Running task pcp... 2026-02-17T04:49:32.438 INFO:teuthology.run_tasks:Running task selinux... 2026-02-17T04:49:32.456 DEBUG:teuthology.orchestra.run.trial100:> sudo service auditd rotate 2026-02-17T04:49:32.485 INFO:teuthology.orchestra.run.trial100.stdout:Rotating logs: 2026-02-17T04:49:32.486 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-17T04:49:32.486 DEBUG:teuthology.orchestra.run.trial100:> /usr/sbin/getenforce 2026-02-17T04:49:32.543 INFO:teuthology.orchestra.run.trial100.stdout:Permissive 2026-02-17T04:49:32.543 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial100.front.sepia.ceph.com': 'permissive'} 2026-02-17T04:49:32.543 DEBUG:teuthology.orchestra.run.trial100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-17T04:49:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-17T04:49:32.606 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-17T04:49:32.606 DEBUG:teuthology.orchestra.run.trial100:> sudo /usr/sbin/setenforce permissive 2026-02-17T04:49:32.670 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-17T04:49:32.682 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-17T04:49:32.695 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-17T04:49:32.697 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial100.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-17T04:52:26.034 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial100.front.sepia.ceph.com')] 2026-02-17T04:52:26.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial100.front.sepia.ceph.com' 2026-02-17T04:52:26.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-17T04:52:26.145 DEBUG:teuthology.orchestra.run.trial100:> true 2026-02-17T04:52:26.223 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial100.front.sepia.ceph.com' 2026-02-17T04:52:26.223 INFO:teuthology.run_tasks:Running task clock... 2026-02-17T04:52:26.235 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-17T04:52:26.236 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-17T04:52:26.236 DEBUG:teuthology.orchestra.run.trial100:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-17T04:52:26.290 INFO:teuthology.orchestra.run.trial100.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-17T04:52:26.301 INFO:teuthology.orchestra.run.trial100.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-17T04:52:26.321 INFO:teuthology.orchestra.run.trial100.stderr:sudo: ntpd: command not found 2026-02-17T04:52:26.330 INFO:teuthology.orchestra.run.trial100.stdout:506 Cannot talk to daemon 2026-02-17T04:52:26.342 INFO:teuthology.orchestra.run.trial100.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-17T04:52:26.353 INFO:teuthology.orchestra.run.trial100.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-17T04:52:26.409 INFO:teuthology.orchestra.run.trial100.stderr:bash: line 1: ntpq: command not found 2026-02-17T04:52:26.411 INFO:teuthology.orchestra.run.trial100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-17T04:52:26.411 INFO:teuthology.orchestra.run.trial100.stdout:=============================================================================== 2026-02-17T04:52:26.412 INFO:teuthology.run_tasks:Running task install... 2026-02-17T04:52:26.430 DEBUG:teuthology.task.install:project ceph 2026-02-17T04:52:26.430 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '325850157d309f65985eea9ec9bb9f1d56726fee'}} 2026-02-17T04:52:26.430 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '325850157d309f65985eea9ec9bb9f1d56726fee'} 2026-02-17T04:52:26.430 INFO:teuthology.task.install:Using flavor: default 2026-02-17T04:52:26.443 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-17T04:52:26.443 INFO:teuthology.task.install:extra packages: [] 2026-02-17T04:52:26.444 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '325850157d309f65985eea9ec9bb9f1d56726fee', 'tag': None, 'wait_for_package': False} 2026-02-17T04:52:26.444 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=325850157d309f65985eea9ec9bb9f1d56726fee 2026-02-17T04:52:26.616 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath18-testing-2026-02-13-0856/325850157d309f65985eea9ec9bb9f1d56726fee/centos/9/flavors/default/ 2026-02-17T04:52:26.617 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5271.g32585015 2026-02-17T04:52:26.748 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath18-testing-2026-02-13-0856/325850157d309f65985eea9ec9bb9f1d56726fee/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath18-testing-2026-02-13-0856/325850157d309f65985eea9ec9bb9f1d56726fee/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath18-testing-2026-02-13-0856/325850157d309f65985eea9ec9bb9f1d56726fee/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-17T04:52:26.748 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:52:26.748 DEBUG:teuthology.orchestra.run.trial100:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-17T04:52:26.774 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-17T04:52:26.774 DEBUG:teuthology.orchestra.run.trial100:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/325850157d309f65985eea9ec9bb9f1d56726fee/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-17T04:52:26.838 DEBUG:teuthology.orchestra.run.trial100:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-17T04:52:26.911 DEBUG:teuthology.orchestra.run.trial100:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-17T04:52:26.975 INFO:teuthology.orchestra.run.trial100.stdout:check_obsoletes = 1 2026-02-17T04:52:26.977 DEBUG:teuthology.orchestra.run.trial100:> sudo yum clean all 2026-02-17T04:52:27.129 INFO:teuthology.orchestra.run.trial100.stdout:41 files removed 2026-02-17T04:52:27.144 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-radosgw 2026-02-17T04:52:27.721 INFO:teuthology.orchestra.run.trial100.stdout:ceph packages for x86_64 217 kB/s | 93 kB 00:00 2026-02-17T04:52:27.893 INFO:teuthology.orchestra.run.trial100.stdout:ceph noarch packages 169 kB/s | 24 kB 00:00 2026-02-17T04:52:28.019 INFO:teuthology.orchestra.run.trial100.stdout:ceph source packages 20 kB/s | 2.1 kB 00:00 2026-02-17T04:52:29.405 INFO:teuthology.orchestra.run.trial100.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.9 MB 00:01 2026-02-17T04:52:32.460 INFO:teuthology.orchestra.run.trial100.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-17T04:52:43.811 INFO:teuthology.orchestra.run.trial100.stdout:CentOS Stream 9 - CRB 890 kB/s | 7.8 MB 00:08 2026-02-17T04:52:44.734 INFO:teuthology.orchestra.run.trial100.stdout:CentOS Stream 9 - Extras packages 105 kB/s | 20 kB 00:00 2026-02-17T04:52:45.028 INFO:teuthology.orchestra.run.trial100.stdout:Extra Packages for Enterprise Linux 90 MB/s | 20 MB 00:00 2026-02-17T04:52:48.266 INFO:teuthology.orchestra.run.trial100.stdout:lab-extras 1.8 MB/s | 50 kB 00:00 2026-02-17T04:52:49.483 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:52:49.484 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:52:49.484 INFO:teuthology.orchestra.run.trial100.stdout: Package Arch Version Repository Size 2026-02-17T04:52:49.484 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:52:49.485 INFO:teuthology.orchestra.run.trial100.stdout:Installing: 2026-02-17T04:52:49.485 INFO:teuthology.orchestra.run.trial100.stdout: ceph-radosgw x86_64 2:20.3.0-5271.g32585015.el9 ceph 13 M 2026-02-17T04:52:49.485 INFO:teuthology.orchestra.run.trial100.stdout:Upgrading: 2026-02-17T04:52:49.485 INFO:teuthology.orchestra.run.trial100.stdout: librados2 x86_64 2:20.3.0-5271.g32585015.el9 ceph 3.5 M 2026-02-17T04:52:49.485 INFO:teuthology.orchestra.run.trial100.stdout: librbd1 x86_64 2:20.3.0-5271.g32585015.el9 ceph 2.8 M 2026-02-17T04:52:49.485 INFO:teuthology.orchestra.run.trial100.stdout:Installing dependencies: 2026-02-17T04:52:49.485 INFO:teuthology.orchestra.run.trial100.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-17T04:52:49.485 INFO:teuthology.orchestra.run.trial100.stdout: ceph-base x86_64 2:20.3.0-5271.g32585015.el9 ceph 5.2 M 2026-02-17T04:52:49.485 INFO:teuthology.orchestra.run.trial100.stdout: ceph-common x86_64 2:20.3.0-5271.g32585015.el9 ceph 24 M 2026-02-17T04:52:49.485 INFO:teuthology.orchestra.run.trial100.stdout: ceph-selinux x86_64 2:20.3.0-5271.g32585015.el9 ceph 25 k 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: libcephfs2 x86_64 2:20.3.0-5271.g32585015.el9 ceph 909 k 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: libradosstriper1 x86_64 2:20.3.0-5271.g32585015.el9 ceph 248 k 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: librgw2 x86_64 2:20.3.0-5271.g32585015.el9 ceph 6.5 M 2026-02-17T04:52:49.486 INFO:teuthology.orchestra.run.trial100.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: python3-ceph-argparse x86_64 2:20.3.0-5271.g32585015.el9 ceph 45 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: python3-ceph-common x86_64 2:20.3.0-5271.g32585015.el9 ceph 185 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: python3-cephfs x86_64 2:20.3.0-5271.g32585015.el9 ceph 204 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: python3-rados x86_64 2:20.3.0-5271.g32585015.el9 ceph 324 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: python3-rbd x86_64 2:20.3.0-5271.g32585015.el9 ceph 306 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: python3-rgw x86_64 2:20.3.0-5271.g32585015.el9 ceph 100 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-17T04:52:49.487 INFO:teuthology.orchestra.run.trial100.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-17T04:52:49.488 INFO:teuthology.orchestra.run.trial100.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-17T04:52:49.488 INFO:teuthology.orchestra.run.trial100.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-17T04:52:49.488 INFO:teuthology.orchestra.run.trial100.stdout:Installing weak dependencies: 2026-02-17T04:52:49.488 INFO:teuthology.orchestra.run.trial100.stdout: libcephfs-daemon x86_64 2:20.3.0-5271.g32585015.el9 ceph 37 k 2026-02-17T04:52:49.488 INFO:teuthology.orchestra.run.trial100.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5271.g32585015.el9 ceph 25 k 2026-02-17T04:52:49.488 INFO:teuthology.orchestra.run.trial100.stdout: nvme-cli x86_64 2.13-1.el9 baseos 1.1 M 2026-02-17T04:52:49.488 INFO:teuthology.orchestra.run.trial100.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-17T04:52:49.488 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:52:49.488 INFO:teuthology.orchestra.run.trial100.stdout:Transaction Summary 2026-02-17T04:52:49.488 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:52:49.488 INFO:teuthology.orchestra.run.trial100.stdout:Install 35 Packages 2026-02-17T04:52:49.489 INFO:teuthology.orchestra.run.trial100.stdout:Upgrade 2 Packages 2026-02-17T04:52:49.489 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:52:49.489 INFO:teuthology.orchestra.run.trial100.stdout:Total download size: 68 M 2026-02-17T04:52:49.489 INFO:teuthology.orchestra.run.trial100.stdout:Downloading Packages: 2026-02-17T04:52:50.022 INFO:teuthology.orchestra.run.trial100.stdout:(1/37): ceph-base-20.3.0-5271.g32585015.el9.x86 16 MB/s | 5.2 MB 00:00 2026-02-17T04:52:50.056 INFO:teuthology.orchestra.run.trial100.stdout:(2/37): ceph-selinux-20.3.0-5271.g32585015.el9. 731 kB/s | 25 kB 00:00 2026-02-17T04:52:50.094 INFO:teuthology.orchestra.run.trial100.stdout:(3/37): libcephfs-daemon-20.3.0-5271.g32585015. 972 kB/s | 37 kB 00:00 2026-02-17T04:52:50.133 INFO:teuthology.orchestra.run.trial100.stdout:(4/37): libcephfs-proxy2-20.3.0-5271.g32585015. 639 kB/s | 25 kB 00:00 2026-02-17T04:52:50.193 INFO:teuthology.orchestra.run.trial100.stdout:(5/37): libcephfs2-20.3.0-5271.g32585015.el9.x8 15 MB/s | 909 kB 00:00 2026-02-17T04:52:50.241 INFO:teuthology.orchestra.run.trial100.stdout:(6/37): libradosstriper1-20.3.0-5271.g32585015. 5.0 MB/s | 248 kB 00:00 2026-02-17T04:52:50.474 INFO:teuthology.orchestra.run.trial100.stdout:(7/37): ceph-radosgw-20.3.0-5271.g32585015.el9. 17 MB/s | 13 MB 00:00 2026-02-17T04:52:50.506 INFO:teuthology.orchestra.run.trial100.stdout:(8/37): python3-ceph-argparse-20.3.0-5271.g3258 1.4 MB/s | 45 kB 00:00 2026-02-17T04:52:50.546 INFO:teuthology.orchestra.run.trial100.stdout:(9/37): python3-ceph-common-20.3.0-5271.g325850 4.6 MB/s | 185 kB 00:00 2026-02-17T04:52:50.558 INFO:teuthology.orchestra.run.trial100.stdout:(10/37): librgw2-20.3.0-5271.g32585015.el9.x86_ 20 MB/s | 6.5 MB 00:00 2026-02-17T04:52:50.571 INFO:teuthology.orchestra.run.trial100.stdout:(11/37): python3-cephfs-20.3.0-5271.g32585015.e 7.8 MB/s | 204 kB 00:00 2026-02-17T04:52:50.587 INFO:teuthology.orchestra.run.trial100.stdout:(12/37): python3-rados-20.3.0-5271.g32585015.el 11 MB/s | 324 kB 00:00 2026-02-17T04:52:50.602 INFO:teuthology.orchestra.run.trial100.stdout:(13/37): python3-rbd-20.3.0-5271.g32585015.el9. 9.9 MB/s | 306 kB 00:00 2026-02-17T04:52:50.613 INFO:teuthology.orchestra.run.trial100.stdout:(14/37): python3-rgw-20.3.0-5271.g32585015.el9. 3.8 MB/s | 100 kB 00:00 2026-02-17T04:52:50.691 INFO:teuthology.orchestra.run.trial100.stdout:(15/37): ceph-common-20.3.0-5271.g32585015.el9. 24 MB/s | 24 MB 00:00 2026-02-17T04:52:50.691 INFO:teuthology.orchestra.run.trial100.stdout:(16/37): mailcap-2.1.49-5.el9.noarch.rpm 370 kB/s | 33 kB 00:00 2026-02-17T04:52:50.733 INFO:teuthology.orchestra.run.trial100.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 2.4 MB/s | 104 kB 00:00 2026-02-17T04:52:50.750 INFO:teuthology.orchestra.run.trial100.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 9.4 MB/s | 164 kB 00:00 2026-02-17T04:52:50.761 INFO:teuthology.orchestra.run.trial100.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 14 MB/s | 160 kB 00:00 2026-02-17T04:52:50.764 INFO:teuthology.orchestra.run.trial100.stdout:(20/37): pciutils-3.7.0-7.el9.x86_64.rpm 1.2 MB/s | 93 kB 00:00 2026-02-17T04:52:50.766 INFO:teuthology.orchestra.run.trial100.stdout:(21/37): nvme-cli-2.13-1.el9.x86_64.rpm 7.1 MB/s | 1.1 MB 00:00 2026-02-17T04:52:50.770 INFO:teuthology.orchestra.run.trial100.stdout:(22/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 5.1 MB/s | 45 kB 00:00 2026-02-17T04:52:50.781 INFO:teuthology.orchestra.run.trial100.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 23 MB/s | 240 kB 00:00 2026-02-17T04:52:50.789 INFO:teuthology.orchestra.run.trial100.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 4.2 MB/s | 37 kB 00:00 2026-02-17T04:52:50.799 INFO:teuthology.orchestra.run.trial100.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 7.2 MB/s | 66 kB 00:00 2026-02-17T04:52:50.813 INFO:teuthology.orchestra.run.trial100.stdout:(26/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.1 MB/s | 292 kB 00:00 2026-02-17T04:52:50.820 INFO:teuthology.orchestra.run.trial100.stdout:(27/37): librdkafka-1.6.1-102.el9.x86_64.rpm 12 MB/s | 662 kB 00:00 2026-02-17T04:52:50.902 INFO:teuthology.orchestra.run.trial100.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 299 kB/s | 25 kB 00:00 2026-02-17T04:52:50.959 INFO:teuthology.orchestra.run.trial100.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 873 kB/s | 49 kB 00:00 2026-02-17T04:52:50.988 INFO:teuthology.orchestra.run.trial100.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 2.3 MB/s | 67 kB 00:00 2026-02-17T04:52:50.993 INFO:teuthology.orchestra.run.trial100.stdout:(31/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.5 MB/s | 308 kB 00:00 2026-02-17T04:52:51.028 INFO:teuthology.orchestra.run.trial100.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 5.4 MB/s | 191 kB 00:00 2026-02-17T04:52:51.074 INFO:teuthology.orchestra.run.trial100.stdout:(33/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 9.5 MB/s | 838 kB 00:00 2026-02-17T04:52:51.148 INFO:teuthology.orchestra.run.trial100.stdout:(34/37): librados2-20.3.0-5271.g32585015.el9.x8 47 MB/s | 3.5 MB 00:00 2026-02-17T04:52:51.151 INFO:teuthology.orchestra.run.trial100.stdout:(35/37): thrift-0.15.0-4.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2026-02-17T04:52:51.201 INFO:teuthology.orchestra.run.trial100.stdout:(36/37): librbd1-20.3.0-5271.g32585015.el9.x86_ 53 MB/s | 2.8 MB 00:00 2026-02-17T04:52:51.240 INFO:teuthology.orchestra.run.trial100.stdout:(37/37): libarrow-9.0.0-15.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2026-02-17T04:52:51.242 INFO:teuthology.orchestra.run.trial100.stdout:-------------------------------------------------------------------------------- 2026-02-17T04:52:51.242 INFO:teuthology.orchestra.run.trial100.stdout:Total 39 MB/s | 68 MB 00:01 2026-02-17T04:52:51.344 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction check 2026-02-17T04:52:51.364 INFO:teuthology.orchestra.run.trial100.stdout:Transaction check succeeded. 2026-02-17T04:52:51.365 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction test 2026-02-17T04:52:51.619 INFO:teuthology.orchestra.run.trial100.stdout:Transaction test succeeded. 2026-02-17T04:52:51.619 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction 2026-02-17T04:52:51.936 INFO:teuthology.orchestra.run.trial100.stdout: Preparing : 1/1 2026-02-17T04:52:52.088 INFO:teuthology.orchestra.run.trial100.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-17T04:52:52.092 INFO:teuthology.orchestra.run.trial100.stdout: Upgrading : librados2-2:20.3.0-5271.g32585015.el9.x86_64 2/39 2026-02-17T04:52:52.141 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: librados2-2:20.3.0-5271.g32585015.el9.x86_64 2/39 2026-02-17T04:52:52.142 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libcephfs2-2:20.3.0-5271.g32585015.el9.x86_64 3/39 2026-02-17T04:52:52.153 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: libcephfs2-2:20.3.0-5271.g32585015.el9.x86_64 3/39 2026-02-17T04:52:52.155 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libcephfs-daemon-2:20.3.0-5271.g32585015.el9.x86_6 4/39 2026-02-17T04:52:52.156 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libcephfs-proxy2-2:20.3.0-5271.g32585015.el9.x86_6 5/39 2026-02-17T04:52:52.183 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5271.g32585015.el9.x86_6 5/39 2026-02-17T04:52:52.186 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-rados-2:20.3.0-5271.g32585015.el9.x86_64 6/39 2026-02-17T04:52:52.193 INFO:teuthology.orchestra.run.trial100.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-17T04:52:52.196 INFO:teuthology.orchestra.run.trial100.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-17T04:52:52.199 INFO:teuthology.orchestra.run.trial100.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-17T04:52:52.203 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-17T04:52:52.215 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-ceph-argparse-2:20.3.0-5271.g32585015.el9. 11/39 2026-02-17T04:52:52.227 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-cephfs-2:20.3.0-5271.g32585015.el9.x86_64 12/39 2026-02-17T04:52:52.228 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libradosstriper1-2:20.3.0-5271.g32585015.el9.x86_6 13/39 2026-02-17T04:52:52.250 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5271.g32585015.el9.x86_6 13/39 2026-02-17T04:52:52.253 INFO:teuthology.orchestra.run.trial100.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-17T04:52:52.283 INFO:teuthology.orchestra.run.trial100.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-17T04:52:52.287 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-17T04:52:52.293 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-17T04:52:52.295 INFO:teuthology.orchestra.run.trial100.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-17T04:52:52.299 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-17T04:52:52.421 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-17T04:52:52.422 INFO:teuthology.orchestra.run.trial100.stdout: Upgrading : librbd1-2:20.3.0-5271.g32585015.el9.x86_64 21/39 2026-02-17T04:52:52.448 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: librbd1-2:20.3.0-5271.g32585015.el9.x86_64 21/39 2026-02-17T04:52:52.450 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-rbd-2:20.3.0-5271.g32585015.el9.x86_64 22/39 2026-02-17T04:52:52.456 INFO:teuthology.orchestra.run.trial100.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-17T04:52:52.739 INFO:teuthology.orchestra.run.trial100.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-17T04:52:52.740 INFO:teuthology.orchestra.run.trial100.stdout: Installing : librgw2-2:20.3.0-5271.g32585015.el9.x86_64 25/39 2026-02-17T04:52:52.756 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: librgw2-2:20.3.0-5271.g32585015.el9.x86_64 25/39 2026-02-17T04:52:52.760 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-rgw-2:20.3.0-5271.g32585015.el9.x86_64 26/39 2026-02-17T04:52:52.765 INFO:teuthology.orchestra.run.trial100.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-17T04:52:52.767 INFO:teuthology.orchestra.run.trial100.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-17T04:52:52.768 INFO:teuthology.orchestra.run.trial100.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-17T04:52:52.787 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-17T04:52:52.901 INFO:teuthology.orchestra.run.trial100.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-17T04:52:52.903 INFO:teuthology.orchestra.run.trial100.stdout: Installing : nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-17T04:52:53.183 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-17T04:52:53.184 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-17T04:52:53.184 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:52:53.228 INFO:teuthology.orchestra.run.trial100.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-17T04:52:53.230 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-ceph-common-2:20.3.0-5271.g32585015.el9.x8 33/39 2026-02-17T04:52:54.325 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-common-2:20.3.0-5271.g32585015.el9.x86_64 34/39 2026-02-17T04:52:54.328 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-common-2:20.3.0-5271.g32585015.el9.x86_64 34/39 2026-02-17T04:52:54.588 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-common-2:20.3.0-5271.g32585015.el9.x86_64 34/39 2026-02-17T04:52:54.591 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-base-2:20.3.0-5271.g32585015.el9.x86_64 35/39 2026-02-17T04:52:54.632 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-base-2:20.3.0-5271.g32585015.el9.x86_64 35/39 2026-02-17T04:52:54.633 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-17T04:52:54.633 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-17T04:52:54.633 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:52:54.636 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-selinux-2:20.3.0-5271.g32585015.el9.x86_64 36/39 2026-02-17T04:52:59.173 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5271.g32585015.el9.x86_64 36/39 2026-02-17T04:52:59.174 INFO:teuthology.orchestra.run.trial100.stdout:skipping the directory /sys 2026-02-17T04:52:59.174 INFO:teuthology.orchestra.run.trial100.stdout:skipping the directory /proc 2026-02-17T04:52:59.174 INFO:teuthology.orchestra.run.trial100.stdout:skipping the directory /mnt 2026-02-17T04:52:59.174 INFO:teuthology.orchestra.run.trial100.stdout:skipping the directory /var/tmp 2026-02-17T04:52:59.174 INFO:teuthology.orchestra.run.trial100.stdout:skipping the directory /home 2026-02-17T04:52:59.174 INFO:teuthology.orchestra.run.trial100.stdout:skipping the directory /root 2026-02-17T04:52:59.174 INFO:teuthology.orchestra.run.trial100.stdout:skipping the directory /tmp 2026-02-17T04:52:59.174 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:52:59.746 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-radosgw-2:20.3.0-5271.g32585015.el9.x86_64 37/39 2026-02-17T04:52:59.770 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5271.g32585015.el9.x86_64 37/39 2026-02-17T04:52:59.770 INFO:teuthology.orchestra.run.trial100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-17T04:52:59.770 INFO:teuthology.orchestra.run.trial100.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-17T04:52:59.770 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-17T04:52:59.770 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-17T04:52:59.770 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:52:59.772 INFO:teuthology.orchestra.run.trial100.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-17T04:52:59.785 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-17T04:52:59.785 INFO:teuthology.orchestra.run.trial100.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-17T04:52:59.842 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-17T04:52:59.842 INFO:teuthology.orchestra.run.trial100.stdout:Creating group 'qat' with GID 992. 2026-02-17T04:52:59.842 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:00.132 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-base-2:20.3.0-5271.g32585015.el9.x86_64 1/39 2026-02-17T04:53:00.132 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-common-2:20.3.0-5271.g32585015.el9.x86_64 2/39 2026-02-17T04:53:00.132 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-radosgw-2:20.3.0-5271.g32585015.el9.x86_64 3/39 2026-02-17T04:53:00.132 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-selinux-2:20.3.0-5271.g32585015.el9.x86_64 4/39 2026-02-17T04:53:00.132 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libcephfs-daemon-2:20.3.0-5271.g32585015.el9.x86_6 5/39 2026-02-17T04:53:00.132 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5271.g32585015.el9.x86_6 6/39 2026-02-17T04:53:00.132 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libcephfs2-2:20.3.0-5271.g32585015.el9.x86_64 7/39 2026-02-17T04:53:00.132 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libradosstriper1-2:20.3.0-5271.g32585015.el9.x86_6 8/39 2026-02-17T04:53:00.133 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : librgw2-2:20.3.0-5271.g32585015.el9.x86_64 9/39 2026-02-17T04:53:00.133 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5271.g32585015.el9. 10/39 2026-02-17T04:53:00.133 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-ceph-common-2:20.3.0-5271.g32585015.el9.x8 11/39 2026-02-17T04:53:00.133 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-cephfs-2:20.3.0-5271.g32585015.el9.x86_64 12/39 2026-02-17T04:53:00.133 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-rados-2:20.3.0-5271.g32585015.el9.x86_64 13/39 2026-02-17T04:53:00.133 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-rbd-2:20.3.0-5271.g32585015.el9.x86_64 14/39 2026-02-17T04:53:00.133 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-rgw-2:20.3.0-5271.g32585015.el9.x86_64 15/39 2026-02-17T04:53:00.133 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : nvme-cli-2.13-1.el9.x86_64 17/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-17T04:53:00.134 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-17T04:53:00.135 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-17T04:53:00.135 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-17T04:53:00.135 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-17T04:53:00.135 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-17T04:53:00.135 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-17T04:53:00.135 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-17T04:53:00.135 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : librados2-2:20.3.0-5271.g32585015.el9.x86_64 36/39 2026-02-17T04:53:00.135 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-17T04:53:00.135 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : librbd1-2:20.3.0-5271.g32585015.el9.x86_64 38/39 2026-02-17T04:53:00.192 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-17T04:53:00.192 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:00.193 INFO:teuthology.orchestra.run.trial100.stdout:Upgraded: 2026-02-17T04:53:00.193 INFO:teuthology.orchestra.run.trial100.stdout: librados2-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.193 INFO:teuthology.orchestra.run.trial100.stdout: librbd1-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.193 INFO:teuthology.orchestra.run.trial100.stdout:Installed: 2026-02-17T04:53:00.193 INFO:teuthology.orchestra.run.trial100.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-17T04:53:00.194 INFO:teuthology.orchestra.run.trial100.stdout: ceph-base-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.194 INFO:teuthology.orchestra.run.trial100.stdout: ceph-common-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.194 INFO:teuthology.orchestra.run.trial100.stdout: ceph-radosgw-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.194 INFO:teuthology.orchestra.run.trial100.stdout: ceph-selinux-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.194 INFO:teuthology.orchestra.run.trial100.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-17T04:53:00.194 INFO:teuthology.orchestra.run.trial100.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-17T04:53:00.195 INFO:teuthology.orchestra.run.trial100.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-17T04:53:00.195 INFO:teuthology.orchestra.run.trial100.stdout: libcephfs-daemon-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.195 INFO:teuthology.orchestra.run.trial100.stdout: libcephfs-proxy2-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.195 INFO:teuthology.orchestra.run.trial100.stdout: libcephfs2-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.195 INFO:teuthology.orchestra.run.trial100.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-17T04:53:00.196 INFO:teuthology.orchestra.run.trial100.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-17T04:53:00.196 INFO:teuthology.orchestra.run.trial100.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-17T04:53:00.196 INFO:teuthology.orchestra.run.trial100.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-17T04:53:00.196 INFO:teuthology.orchestra.run.trial100.stdout: libradosstriper1-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.196 INFO:teuthology.orchestra.run.trial100.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-17T04:53:00.196 INFO:teuthology.orchestra.run.trial100.stdout: librgw2-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.197 INFO:teuthology.orchestra.run.trial100.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-17T04:53:00.197 INFO:teuthology.orchestra.run.trial100.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-17T04:53:00.197 INFO:teuthology.orchestra.run.trial100.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-17T04:53:00.197 INFO:teuthology.orchestra.run.trial100.stdout: nvme-cli-2.13-1.el9.x86_64 2026-02-17T04:53:00.197 INFO:teuthology.orchestra.run.trial100.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-17T04:53:00.198 INFO:teuthology.orchestra.run.trial100.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-17T04:53:00.198 INFO:teuthology.orchestra.run.trial100.stdout: python3-ceph-argparse-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.198 INFO:teuthology.orchestra.run.trial100.stdout: python3-ceph-common-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.198 INFO:teuthology.orchestra.run.trial100.stdout: python3-cephfs-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.198 INFO:teuthology.orchestra.run.trial100.stdout: python3-rados-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.198 INFO:teuthology.orchestra.run.trial100.stdout: python3-rbd-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.199 INFO:teuthology.orchestra.run.trial100.stdout: python3-rgw-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:00.199 INFO:teuthology.orchestra.run.trial100.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-17T04:53:00.199 INFO:teuthology.orchestra.run.trial100.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-17T04:53:00.199 INFO:teuthology.orchestra.run.trial100.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-17T04:53:00.199 INFO:teuthology.orchestra.run.trial100.stdout: re2-1:20211101-20.el9.x86_64 2026-02-17T04:53:00.199 INFO:teuthology.orchestra.run.trial100.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-17T04:53:00.200 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:00.200 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:00.236 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-test 2026-02-17T04:53:00.550 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:12 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:00.635 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:00.635 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:00.635 INFO:teuthology.orchestra.run.trial100.stdout: Package Arch Version Repository Size 2026-02-17T04:53:00.635 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout:Installing: 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: ceph-test x86_64 2:20.3.0-5271.g32585015.el9 ceph 78 M 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout:Installing dependencies: 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: libcephsqlite x86_64 2:20.3.0-5271.g32585015.el9 ceph 159 k 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout:Transaction Summary 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout:Install 10 Packages 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout:Total download size: 82 M 2026-02-17T04:53:00.636 INFO:teuthology.orchestra.run.trial100.stdout:Installed size: 339 M 2026-02-17T04:53:00.637 INFO:teuthology.orchestra.run.trial100.stdout:Downloading Packages: 2026-02-17T04:53:00.963 INFO:teuthology.orchestra.run.trial100.stdout:(1/10): hwloc-libs-2.4.1-6.el9.x86_64.rpm 26 MB/s | 2.1 MB 00:00 2026-02-17T04:53:00.964 INFO:teuthology.orchestra.run.trial100.stdout:(2/10): libcephsqlite-20.3.0-5271.g32585015.el9 1.9 MB/s | 159 kB 00:00 2026-02-17T04:53:00.971 INFO:teuthology.orchestra.run.trial100.stdout:(3/10): libatomic-11.5.0-14.el9.x86_64.rpm 2.7 MB/s | 23 kB 00:00 2026-02-17T04:53:00.995 INFO:teuthology.orchestra.run.trial100.stdout:(4/10): libxslt-1.1.34-12.el9.x86_64.rpm 9.4 MB/s | 233 kB 00:00 2026-02-17T04:53:00.999 INFO:teuthology.orchestra.run.trial100.stdout:(5/10): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.6 MB/s | 94 kB 00:00 2026-02-17T04:53:01.019 INFO:teuthology.orchestra.run.trial100.stdout:(6/10): socat-1.7.4.1-8.el9.x86_64.rpm 14 MB/s | 303 kB 00:00 2026-02-17T04:53:01.024 INFO:teuthology.orchestra.run.trial100.stdout:(7/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 13 MB/s | 64 kB 00:00 2026-02-17T04:53:01.027 INFO:teuthology.orchestra.run.trial100.stdout:(8/10): protobuf-3.14.0-17.el9.x86_64.rpm 32 MB/s | 1.0 MB 00:00 2026-02-17T04:53:01.045 INFO:teuthology.orchestra.run.trial100.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 5.8 MB/s | 122 kB 00:00 2026-02-17T04:53:02.675 INFO:teuthology.orchestra.run.trial100.stdout:(10/10): ceph-test-20.3.0-5271.g32585015.el9.x8 44 MB/s | 78 MB 00:01 2026-02-17T04:53:02.676 INFO:teuthology.orchestra.run.trial100.stdout:-------------------------------------------------------------------------------- 2026-02-17T04:53:02.676 INFO:teuthology.orchestra.run.trial100.stdout:Total 40 MB/s | 82 MB 00:02 2026-02-17T04:53:02.741 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction check 2026-02-17T04:53:02.753 INFO:teuthology.orchestra.run.trial100.stdout:Transaction check succeeded. 2026-02-17T04:53:02.753 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction test 2026-02-17T04:53:03.035 INFO:teuthology.orchestra.run.trial100.stdout:Transaction test succeeded. 2026-02-17T04:53:03.036 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction 2026-02-17T04:53:03.344 INFO:teuthology.orchestra.run.trial100.stdout: Preparing : 1/1 2026-02-17T04:53:03.353 INFO:teuthology.orchestra.run.trial100.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2026-02-17T04:53:03.361 INFO:teuthology.orchestra.run.trial100.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/10 2026-02-17T04:53:03.366 INFO:teuthology.orchestra.run.trial100.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/10 2026-02-17T04:53:03.372 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/10 2026-02-17T04:53:03.378 INFO:teuthology.orchestra.run.trial100.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2026-02-17T04:53:03.380 INFO:teuthology.orchestra.run.trial100.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 6/10 2026-02-17T04:53:03.390 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 7/10 2026-02-17T04:53:03.398 INFO:teuthology.orchestra.run.trial100.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 8/10 2026-02-17T04:53:03.399 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libcephsqlite-2:20.3.0-5271.g32585015.el9.x86_64 9/10 2026-02-17T04:53:06.841 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5271.g32585015.el9.x86_64 9/10 2026-02-17T04:53:06.860 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-test-2:20.3.0-5271.g32585015.el9.x86_64 10/10 2026-02-17T04:53:07.207 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-test-2:20.3.0-5271.g32585015.el9.x86_64 10/10 2026-02-17T04:53:07.208 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-test-2:20.3.0-5271.g32585015.el9.x86_64 1/10 2026-02-17T04:53:07.208 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libcephsqlite-2:20.3.0-5271.g32585015.el9.x86_64 2/10 2026-02-17T04:53:07.208 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/10 2026-02-17T04:53:07.208 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/10 2026-02-17T04:53:07.208 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/10 2026-02-17T04:53:07.208 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 6/10 2026-02-17T04:53:07.208 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 7/10 2026-02-17T04:53:07.208 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 8/10 2026-02-17T04:53:07.208 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 9/10 2026-02-17T04:53:07.261 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2026-02-17T04:53:07.262 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:07.262 INFO:teuthology.orchestra.run.trial100.stdout:Installed: 2026-02-17T04:53:07.262 INFO:teuthology.orchestra.run.trial100.stdout: ceph-test-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:07.262 INFO:teuthology.orchestra.run.trial100.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-17T04:53:07.263 INFO:teuthology.orchestra.run.trial100.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-17T04:53:07.263 INFO:teuthology.orchestra.run.trial100.stdout: libcephsqlite-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:07.263 INFO:teuthology.orchestra.run.trial100.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-17T04:53:07.263 INFO:teuthology.orchestra.run.trial100.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-17T04:53:07.263 INFO:teuthology.orchestra.run.trial100.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-17T04:53:07.263 INFO:teuthology.orchestra.run.trial100.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-17T04:53:07.264 INFO:teuthology.orchestra.run.trial100.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-17T04:53:07.264 INFO:teuthology.orchestra.run.trial100.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-17T04:53:07.264 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:07.264 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:07.300 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph 2026-02-17T04:53:07.613 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:19 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:07.698 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:07.701 INFO:teuthology.orchestra.run.trial100.stdout:======================================================================================= 2026-02-17T04:53:07.701 INFO:teuthology.orchestra.run.trial100.stdout: Package Arch Version Repository Size 2026-02-17T04:53:07.702 INFO:teuthology.orchestra.run.trial100.stdout:======================================================================================= 2026-02-17T04:53:07.702 INFO:teuthology.orchestra.run.trial100.stdout:Installing: 2026-02-17T04:53:07.702 INFO:teuthology.orchestra.run.trial100.stdout: ceph x86_64 2:20.3.0-5271.g32585015.el9 ceph 6.5 k 2026-02-17T04:53:07.702 INFO:teuthology.orchestra.run.trial100.stdout:Installing dependencies: 2026-02-17T04:53:07.702 INFO:teuthology.orchestra.run.trial100.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-17T04:53:07.702 INFO:teuthology.orchestra.run.trial100.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5271.g32585015.el9 ceph-noarch 57 k 2026-02-17T04:53:07.702 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mds x86_64 2:20.3.0-5271.g32585015.el9 ceph 2.4 M 2026-02-17T04:53:07.702 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr x86_64 2:20.3.0-5271.g32585015.el9 ceph 936 k 2026-02-17T04:53:07.702 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5271.g32585015.el9 ceph-noarch 297 k 2026-02-17T04:53:07.702 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mon x86_64 2:20.3.0-5271.g32585015.el9 ceph 5.0 M 2026-02-17T04:53:07.702 INFO:teuthology.orchestra.run.trial100.stdout: ceph-osd x86_64 2:20.3.0-5271.g32585015.el9 ceph 167 k 2026-02-17T04:53:07.703 INFO:teuthology.orchestra.run.trial100.stdout: ceph-osd-classic x86_64 2:20.3.0-5271.g32585015.el9 ceph 16 M 2026-02-17T04:53:07.703 INFO:teuthology.orchestra.run.trial100.stdout: ceph-osd-crimson x86_64 2:20.3.0-5271.g32585015.el9 ceph 19 M 2026-02-17T04:53:07.703 INFO:teuthology.orchestra.run.trial100.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5271.g32585015.el9 ceph-noarch 17 k 2026-02-17T04:53:07.703 INFO:teuthology.orchestra.run.trial100.stdout: cephadm noarch 2:20.3.0-5271.g32585015.el9 ceph-noarch 1.0 M 2026-02-17T04:53:07.703 INFO:teuthology.orchestra.run.trial100.stdout: cryptsetup x86_64 2.8.1-2.el9 baseos 350 k 2026-02-17T04:53:07.703 INFO:teuthology.orchestra.run.trial100.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-17T04:53:07.703 INFO:teuthology.orchestra.run.trial100.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-17T04:53:07.703 INFO:teuthology.orchestra.run.trial100.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-17T04:53:07.703 INFO:teuthology.orchestra.run.trial100.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-17T04:53:07.704 INFO:teuthology.orchestra.run.trial100.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-devel x86_64 3.9.25-2.el9 appstream 244 k 2026-02-17T04:53:07.705 INFO:teuthology.orchestra.run.trial100.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-17T04:53:07.706 INFO:teuthology.orchestra.run.trial100.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-17T04:53:07.707 INFO:teuthology.orchestra.run.trial100.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-17T04:53:07.708 INFO:teuthology.orchestra.run.trial100.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout:Installing weak dependencies: 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5271.g32585015.el9 ceph-noarch 184 k 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5271.g32585015.el9 ceph-noarch 51 M 2026-02-17T04:53:07.709 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5271.g32585015.el9 ceph-noarch 7.4 M 2026-02-17T04:53:07.710 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5271.g32585015.el9 ceph-noarch 22 k 2026-02-17T04:53:07.710 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-rook noarch 2:20.3.0-5271.g32585015.el9 ceph-noarch 50 k 2026-02-17T04:53:07.710 INFO:teuthology.orchestra.run.trial100.stdout: ceph-volume noarch 2:20.3.0-5271.g32585015.el9 ceph-noarch 299 k 2026-02-17T04:53:07.710 INFO:teuthology.orchestra.run.trial100.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-17T04:53:07.710 INFO:teuthology.orchestra.run.trial100.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-17T04:53:07.710 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:07.710 INFO:teuthology.orchestra.run.trial100.stdout:Transaction Summary 2026-02-17T04:53:07.710 INFO:teuthology.orchestra.run.trial100.stdout:======================================================================================= 2026-02-17T04:53:07.710 INFO:teuthology.orchestra.run.trial100.stdout:Install 96 Packages 2026-02-17T04:53:07.710 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:07.711 INFO:teuthology.orchestra.run.trial100.stdout:Total download size: 158 M 2026-02-17T04:53:07.711 INFO:teuthology.orchestra.run.trial100.stdout:Installed size: 732 M 2026-02-17T04:53:07.711 INFO:teuthology.orchestra.run.trial100.stdout:Downloading Packages: 2026-02-17T04:53:08.363 INFO:teuthology.orchestra.run.trial100.stdout:(1/96): ceph-20.3.0-5271.g32585015.el9.x86_64.r 128 kB/s | 6.5 kB 00:00 2026-02-17T04:53:08.448 INFO:teuthology.orchestra.run.trial100.stdout:(2/96): ceph-mgr-20.3.0-5271.g32585015.el9.x86_ 6.7 MB/s | 936 kB 00:00 2026-02-17T04:53:08.479 INFO:teuthology.orchestra.run.trial100.stdout:(3/96): ceph-mds-20.3.0-5271.g32585015.el9.x86_ 14 MB/s | 2.4 MB 00:00 2026-02-17T04:53:08.482 INFO:teuthology.orchestra.run.trial100.stdout:(4/96): ceph-osd-20.3.0-5271.g32585015.el9.x86_ 4.8 MB/s | 167 kB 00:00 2026-02-17T04:53:08.607 INFO:teuthology.orchestra.run.trial100.stdout:(5/96): ceph-mon-20.3.0-5271.g32585015.el9.x86_ 20 MB/s | 5.0 MB 00:00 2026-02-17T04:53:08.645 INFO:teuthology.orchestra.run.trial100.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5271.g32 1.5 MB/s | 57 kB 00:00 2026-02-17T04:53:08.687 INFO:teuthology.orchestra.run.trial100.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5271.g32585015. 4.2 MB/s | 184 kB 00:00 2026-02-17T04:53:09.153 INFO:teuthology.orchestra.run.trial100.stdout:(8/96): ceph-osd-classic-20.3.0-5271.g32585015. 23 MB/s | 16 MB 00:00 2026-02-17T04:53:09.507 INFO:teuthology.orchestra.run.trial100.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-52 21 MB/s | 7.4 MB 00:00 2026-02-17T04:53:09.539 INFO:teuthology.orchestra.run.trial100.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5271.g325850 724 kB/s | 22 kB 00:00 2026-02-17T04:53:09.578 INFO:teuthology.orchestra.run.trial100.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5271.g325 7.4 MB/s | 297 kB 00:00 2026-02-17T04:53:09.612 INFO:teuthology.orchestra.run.trial100.stdout:(12/96): ceph-mgr-rook-20.3.0-5271.g32585015.el 1.4 MB/s | 50 kB 00:00 2026-02-17T04:53:09.658 INFO:teuthology.orchestra.run.trial100.stdout:(13/96): ceph-osd-crimson-20.3.0-5271.g32585015 16 MB/s | 19 MB 00:01 2026-02-17T04:53:09.659 INFO:teuthology.orchestra.run.trial100.stdout:(14/96): ceph-prometheus-alerts-20.3.0-5271.g32 378 kB/s | 17 kB 00:00 2026-02-17T04:53:09.679 INFO:teuthology.orchestra.run.trial100.stdout:(15/96): ceph-volume-20.3.0-5271.g32585015.el9. 14 MB/s | 299 kB 00:00 2026-02-17T04:53:09.702 INFO:teuthology.orchestra.run.trial100.stdout:(16/96): cephadm-20.3.0-5271.g32585015.el9.noar 23 MB/s | 1.0 MB 00:00 2026-02-17T04:53:10.241 INFO:teuthology.orchestra.run.trial100.stdout:(17/96): ceph-mgr-dashboard-20.3.0-5271.g325850 33 MB/s | 51 MB 00:01 2026-02-17T04:53:10.241 INFO:teuthology.orchestra.run.trial100.stdout:(18/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 75 kB/s | 40 kB 00:00 2026-02-17T04:53:10.669 INFO:teuthology.orchestra.run.trial100.stdout:(19/96): libconfig-1.7.2-9.el9.x86_64.rpm 168 kB/s | 72 kB 00:00 2026-02-17T04:53:10.758 INFO:teuthology.orchestra.run.trial100.stdout:(20/96): cryptsetup-2.8.1-2.el9.x86_64.rpm 325 kB/s | 350 kB 00:01 2026-02-17T04:53:11.106 INFO:teuthology.orchestra.run.trial100.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 728 kB/s | 253 kB 00:00 2026-02-17T04:53:11.423 INFO:teuthology.orchestra.run.trial100.stdout:(22/96): libquadmath-11.5.0-14.el9.x86_64.rpm 245 kB/s | 184 kB 00:00 2026-02-17T04:53:11.772 INFO:teuthology.orchestra.run.trial100.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 305 kB/s | 106 kB 00:00 2026-02-17T04:53:12.119 INFO:teuthology.orchestra.run.trial100.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 389 kB/s | 135 kB 00:00 2026-02-17T04:53:12.479 INFO:teuthology.orchestra.run.trial100.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 419 kB/s | 150 kB 00:00 2026-02-17T04:53:12.682 INFO:teuthology.orchestra.run.trial100.stdout:(26/96): python3-cryptography-36.0.1-5.el9.x86_ 809 kB/s | 1.2 MB 00:01 2026-02-17T04:53:12.693 INFO:teuthology.orchestra.run.trial100.stdout:(27/96): libgfortran-11.5.0-14.el9.x86_64.rpm 324 kB/s | 794 kB 00:02 2026-02-17T04:53:12.745 INFO:teuthology.orchestra.run.trial100.stdout:(28/96): python3-requests-2.25.1-10.el9.noarch. 474 kB/s | 126 kB 00:00 2026-02-17T04:53:13.034 INFO:teuthology.orchestra.run.trial100.stdout:(29/96): python3-urllib3-1.26.5-7.el9.noarch.rp 619 kB/s | 218 kB 00:00 2026-02-17T04:53:13.041 INFO:teuthology.orchestra.run.trial100.stdout:(30/96): unzip-6.0-59.el9.x86_64.rpm 524 kB/s | 182 kB 00:00 2026-02-17T04:53:13.065 INFO:teuthology.orchestra.run.trial100.stdout:(31/96): flexiblas-3.0.4-9.el9.x86_64.rpm 960 kB/s | 30 kB 00:00 2026-02-17T04:53:13.074 INFO:teuthology.orchestra.run.trial100.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 1.7 MB/s | 15 kB 00:00 2026-02-17T04:53:13.105 INFO:teuthology.orchestra.run.trial100.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 7.8 MB/s | 246 kB 00:00 2026-02-17T04:53:13.116 INFO:teuthology.orchestra.run.trial100.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 16 MB/s | 188 kB 00:00 2026-02-17T04:53:13.121 INFO:teuthology.orchestra.run.trial100.stdout:(35/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 38 MB/s | 3.0 MB 00:00 2026-02-17T04:53:13.125 INFO:teuthology.orchestra.run.trial100.stdout:(36/96): openblas-0.3.29-1.el9.x86_64.rpm 4.7 MB/s | 42 kB 00:00 2026-02-17T04:53:13.149 INFO:teuthology.orchestra.run.trial100.stdout:(37/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 183 MB/s | 5.3 MB 00:00 2026-02-17T04:53:13.161 INFO:teuthology.orchestra.run.trial100.stdout:(38/96): python3-devel-3.9.25-2.el9.x86_64.rpm 21 MB/s | 244 kB 00:00 2026-02-17T04:53:13.178 INFO:teuthology.orchestra.run.trial100.stdout:(39/96): python3-babel-2.9.1-2.el9.noarch.rpm 113 MB/s | 6.0 MB 00:00 2026-02-17T04:53:13.179 INFO:teuthology.orchestra.run.trial100.stdout:(40/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 14 MB/s | 249 kB 00:00 2026-02-17T04:53:13.182 INFO:teuthology.orchestra.run.trial100.stdout:(41/96): zip-3.0-35.el9.x86_64.rpm 608 kB/s | 266 kB 00:00 2026-02-17T04:53:13.188 INFO:teuthology.orchestra.run.trial100.stdout:(42/96): python3-jmespath-1.0.1-1.el9.noarch.rp 4.9 MB/s | 48 kB 00:00 2026-02-17T04:53:13.190 INFO:teuthology.orchestra.run.trial100.stdout:(43/96): python3-libstoragemgmt-1.10.1-1.el9.x8 16 MB/s | 177 kB 00:00 2026-02-17T04:53:13.197 INFO:teuthology.orchestra.run.trial100.stdout:(44/96): python3-markupsafe-1.1.1-12.el9.x86_64 3.7 MB/s | 35 kB 00:00 2026-02-17T04:53:13.207 INFO:teuthology.orchestra.run.trial100.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 42 MB/s | 442 kB 00:00 2026-02-17T04:53:13.218 INFO:teuthology.orchestra.run.trial100.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 7.4 MB/s | 77 kB 00:00 2026-02-17T04:53:13.228 INFO:teuthology.orchestra.run.trial100.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 25 MB/s | 267 kB 00:00 2026-02-17T04:53:13.238 INFO:teuthology.orchestra.run.trial100.stdout:(48/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 128 MB/s | 6.1 MB 00:00 2026-02-17T04:53:13.240 INFO:teuthology.orchestra.run.trial100.stdout:(49/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 13 MB/s | 159 kB 00:00 2026-02-17T04:53:13.250 INFO:teuthology.orchestra.run.trial100.stdout:(50/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 18 MB/s | 1.2 MB 00:00 2026-02-17T04:53:13.250 INFO:teuthology.orchestra.run.trial100.stdout:(51/96): python3-requests-oauthlib-1.3.0-12.el9 4.9 MB/s | 54 kB 00:00 2026-02-17T04:53:13.251 INFO:teuthology.orchestra.run.trial100.stdout:(52/96): python3-pyasn1-modules-0.4.8-6.el9.noa 21 MB/s | 279 kB 00:00 2026-02-17T04:53:13.260 INFO:teuthology.orchestra.run.trial100.stdout:(53/96): python3-toml-0.10.2-6.el9.noarch.rpm 4.3 MB/s | 42 kB 00:00 2026-02-17T04:53:13.262 INFO:teuthology.orchestra.run.trial100.stdout:(54/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 17 MB/s | 189 kB 00:00 2026-02-17T04:53:13.269 INFO:teuthology.orchestra.run.trial100.stdout:(55/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 10 MB/s | 90 kB 00:00 2026-02-17T04:53:13.282 INFO:teuthology.orchestra.run.trial100.stdout:(56/96): lua-devel-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 22 kB 00:00 2026-02-17T04:53:13.299 INFO:teuthology.orchestra.run.trial100.stdout:(57/96): protobuf-compiler-3.14.0-17.el9.x86_64 27 MB/s | 862 kB 00:00 2026-02-17T04:53:13.347 INFO:teuthology.orchestra.run.trial100.stdout:(58/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 199 MB/s | 19 MB 00:00 2026-02-17T04:53:13.419 INFO:teuthology.orchestra.run.trial100.stdout:(59/96): grpc-data-1.46.7-10.el9.noarch.rpm 164 kB/s | 19 kB 00:00 2026-02-17T04:53:13.483 INFO:teuthology.orchestra.run.trial100.stdout:(60/96): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2026-02-17T04:53:13.511 INFO:teuthology.orchestra.run.trial100.stdout:(61/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.4 MB/s | 551 kB 00:00 2026-02-17T04:53:13.534 INFO:teuthology.orchestra.run.trial100.stdout:(62/96): python3-asyncssh-2.13.2-5.el9.noarch.r 4.7 MB/s | 548 kB 00:00 2026-02-17T04:53:13.534 INFO:teuthology.orchestra.run.trial100.stdout:(63/96): python3-autocommand-2.2.2-8.el9.noarch 575 kB/s | 29 kB 00:00 2026-02-17T04:53:13.535 INFO:teuthology.orchestra.run.trial100.stdout:(64/96): python3-backports-tarfile-1.2.0-1.el9. 2.4 MB/s | 60 kB 00:00 2026-02-17T04:53:13.558 INFO:teuthology.orchestra.run.trial100.stdout:(65/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2026-02-17T04:53:13.559 INFO:teuthology.orchestra.run.trial100.stdout:(66/96): python3-cachetools-4.2.4-1.el9.noarch. 1.3 MB/s | 32 kB 00:00 2026-02-17T04:53:13.559 INFO:teuthology.orchestra.run.trial100.stdout:(67/96): python3-certifi-2023.05.07-4.el9.noarc 585 kB/s | 14 kB 00:00 2026-02-17T04:53:13.582 INFO:teuthology.orchestra.run.trial100.stdout:(68/96): python3-cheroot-10.0.1-4.el9.noarch.rp 7.2 MB/s | 173 kB 00:00 2026-02-17T04:53:13.585 INFO:teuthology.orchestra.run.trial100.stdout:(69/96): python3-cherrypy-18.6.1-2.el9.noarch.r 13 MB/s | 358 kB 00:00 2026-02-17T04:53:13.588 INFO:teuthology.orchestra.run.trial100.stdout:(70/96): python3-google-auth-2.45.0-1.el9.noarc 8.7 MB/s | 254 kB 00:00 2026-02-17T04:53:13.630 INFO:teuthology.orchestra.run.trial100.stdout:(71/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 42 MB/s | 2.0 MB 00:00 2026-02-17T04:53:13.631 INFO:teuthology.orchestra.run.trial100.stdout:(72/96): python3-grpcio-tools-1.46.7-10.el9.x86 3.1 MB/s | 144 kB 00:00 2026-02-17T04:53:13.632 INFO:teuthology.orchestra.run.trial100.stdout:(73/96): python3-influxdb-5.3.1-1.el9.noarch.rp 3.1 MB/s | 139 kB 00:00 2026-02-17T04:53:13.657 INFO:teuthology.orchestra.run.trial100.stdout:(74/96): python3-isodate-0.6.1-3.el9.noarch.rpm 2.2 MB/s | 56 kB 00:00 2026-02-17T04:53:13.657 INFO:teuthology.orchestra.run.trial100.stdout:(75/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 431 kB/s | 11 kB 00:00 2026-02-17T04:53:13.657 INFO:teuthology.orchestra.run.trial100.stdout:(76/96): python3-jaraco-classes-3.2.1-5.el9.noa 707 kB/s | 18 kB 00:00 2026-02-17T04:53:13.680 INFO:teuthology.orchestra.run.trial100.stdout:(77/96): python3-jaraco-collections-3.0.0-8.el9 1.0 MB/s | 23 kB 00:00 2026-02-17T04:53:13.680 INFO:teuthology.orchestra.run.trial100.stdout:(78/96): python3-jaraco-context-6.0.1-3.el9.noa 842 kB/s | 20 kB 00:00 2026-02-17T04:53:13.681 INFO:teuthology.orchestra.run.trial100.stdout:(79/96): python3-jaraco-functools-3.5.0-2.el9.n 825 kB/s | 19 kB 00:00 2026-02-17T04:53:13.705 INFO:teuthology.orchestra.run.trial100.stdout:(80/96): python3-jaraco-text-4.0.0-2.el9.noarch 1.1 MB/s | 26 kB 00:00 2026-02-17T04:53:13.711 INFO:teuthology.orchestra.run.trial100.stdout:(81/96): python3-kubernetes-26.1.0-3.el9.noarch 34 MB/s | 1.0 MB 00:00 2026-02-17T04:53:13.711 INFO:teuthology.orchestra.run.trial100.stdout:(82/96): python3-more-itertools-8.12.0-2.el9.no 2.6 MB/s | 79 kB 00:00 2026-02-17T04:53:13.730 INFO:teuthology.orchestra.run.trial100.stdout:(83/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 3.5 MB/s | 86 kB 00:00 2026-02-17T04:53:13.737 INFO:teuthology.orchestra.run.trial100.stdout:(84/96): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2026-02-17T04:53:13.737 INFO:teuthology.orchestra.run.trial100.stdout:(85/96): python3-portend-3.1.0-2.el9.noarch.rpm 620 kB/s | 16 kB 00:00 2026-02-17T04:53:13.755 INFO:teuthology.orchestra.run.trial100.stdout:(86/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2026-02-17T04:53:13.760 INFO:teuthology.orchestra.run.trial100.stdout:(87/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.3 MB/s | 31 kB 00:00 2026-02-17T04:53:13.761 INFO:teuthology.orchestra.run.trial100.stdout:(88/96): python3-routes-2.5.1-5.el9.noarch.rpm 8.0 MB/s | 188 kB 00:00 2026-02-17T04:53:13.779 INFO:teuthology.orchestra.run.trial100.stdout:(89/96): python3-rsa-4.9-2.el9.noarch.rpm 2.4 MB/s | 59 kB 00:00 2026-02-17T04:53:13.785 INFO:teuthology.orchestra.run.trial100.stdout:(90/96): python3-saml-1.16.0-1.el9.noarch.rpm 4.8 MB/s | 125 kB 00:00 2026-02-17T04:53:13.787 INFO:teuthology.orchestra.run.trial100.stdout:(91/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.3 MB/s | 36 kB 00:00 2026-02-17T04:53:13.804 INFO:teuthology.orchestra.run.trial100.stdout:(92/96): python3-typing-extensions-4.15.0-1.el9 3.4 MB/s | 86 kB 00:00 2026-02-17T04:53:13.809 INFO:teuthology.orchestra.run.trial100.stdout:(93/96): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2026-02-17T04:53:13.810 INFO:teuthology.orchestra.run.trial100.stdout:(94/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 2.0 MB/s | 48 kB 00:00 2026-02-17T04:53:13.829 INFO:teuthology.orchestra.run.trial100.stdout:(95/96): python3-xmltodict-0.12.0-15.el9.noarch 869 kB/s | 22 kB 00:00 2026-02-17T04:53:13.832 INFO:teuthology.orchestra.run.trial100.stdout:(96/96): python3-zc-lockfile-2.0-10.el9.noarch. 847 kB/s | 20 kB 00:00 2026-02-17T04:53:13.836 INFO:teuthology.orchestra.run.trial100.stdout:-------------------------------------------------------------------------------- 2026-02-17T04:53:13.836 INFO:teuthology.orchestra.run.trial100.stdout:Total 26 MB/s | 158 MB 00:06 2026-02-17T04:53:14.265 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction check 2026-02-17T04:53:14.304 INFO:teuthology.orchestra.run.trial100.stdout:Transaction check succeeded. 2026-02-17T04:53:14.304 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction test 2026-02-17T04:53:14.902 INFO:teuthology.orchestra.run.trial100.stdout:Transaction test succeeded. 2026-02-17T04:53:14.902 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction 2026-02-17T04:53:15.524 INFO:teuthology.orchestra.run.trial100.stdout: Preparing : 1/1 2026-02-17T04:53:15.526 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-17T04:53:15.530 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-17T04:53:15.552 INFO:teuthology.orchestra.run.trial100.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-17T04:53:15.585 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-17T04:53:15.591 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-17T04:53:15.594 INFO:teuthology.orchestra.run.trial100.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-17T04:53:15.598 INFO:teuthology.orchestra.run.trial100.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-17T04:53:15.621 INFO:teuthology.orchestra.run.trial100.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-17T04:53:15.634 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-17T04:53:15.637 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-17T04:53:15.643 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-17T04:53:15.645 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-17T04:53:15.656 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-17T04:53:15.662 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-17T04:53:15.686 INFO:teuthology.orchestra.run.trial100.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-17T04:53:15.689 INFO:teuthology.orchestra.run.trial100.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-17T04:53:15.740 INFO:teuthology.orchestra.run.trial100.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-17T04:53:15.752 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-17T04:53:15.755 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-17T04:53:15.759 INFO:teuthology.orchestra.run.trial100.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-17T04:53:15.765 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-17T04:53:15.769 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-17T04:53:15.773 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-17T04:53:15.787 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-17T04:53:15.791 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-17T04:53:15.796 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-17T04:53:15.808 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-17T04:53:15.817 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-17T04:53:15.823 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-17T04:53:15.843 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-17T04:53:15.854 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-17T04:53:15.869 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-17T04:53:15.874 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-17T04:53:15.882 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-17T04:53:15.922 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-17T04:53:16.247 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-17T04:53:16.257 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-17T04:53:16.261 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-17T04:53:16.266 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-17T04:53:16.268 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-17T04:53:16.296 INFO:teuthology.orchestra.run.trial100.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-17T04:53:16.340 INFO:teuthology.orchestra.run.trial100.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-17T04:53:16.348 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-17T04:53:16.354 INFO:teuthology.orchestra.run.trial100.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-17T04:53:16.358 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-17T04:53:16.363 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-17T04:53:16.367 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-17T04:53:16.374 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-17T04:53:16.378 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-17T04:53:16.405 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-17T04:53:16.415 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-17T04:53:16.421 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-17T04:53:16.428 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-17T04:53:16.462 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-17T04:53:16.687 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-devel-3.9.25-2.el9.x86_64 55/96 2026-02-17T04:53:16.711 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-17T04:53:16.715 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-17T04:53:16.776 INFO:teuthology.orchestra.run.trial100.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-17T04:53:16.778 INFO:teuthology.orchestra.run.trial100.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-17T04:53:16.800 INFO:teuthology.orchestra.run.trial100.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-17T04:53:17.125 INFO:teuthology.orchestra.run.trial100.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-17T04:53:17.198 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-17T04:53:17.870 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-17T04:53:17.887 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-17T04:53:17.897 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-17T04:53:17.903 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-17T04:53:17.917 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-17T04:53:17.932 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-17T04:53:18.008 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-17T04:53:18.017 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-17T04:53:18.040 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-17T04:53:18.070 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-17T04:53:18.121 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-17T04:53:18.127 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-17T04:53:18.130 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-17T04:53:18.131 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-17T04:53:18.150 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-17T04:53:18.150 INFO:teuthology.orchestra.run.trial100.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-17T04:53:18.150 INFO:teuthology.orchestra.run.trial100.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-17T04:53:18.150 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:18.158 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-17T04:53:18.185 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-17T04:53:18.186 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-17T04:53:18.186 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:18.221 INFO:teuthology.orchestra.run.trial100.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-17T04:53:18.842 INFO:teuthology.orchestra.run.trial100.stdout: Installing : cryptsetup-2.8.1-2.el9.x86_64 79/96 2026-02-17T04:53:19.599 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-osd-classic-2:20.3.0-5271.g32585015.el9.x86_6 80/96 2026-02-17T04:53:19.663 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-osd-crimson-2:20.3.0-5271.g32585015.el9.x86_6 81/96 2026-02-17T04:53:19.665 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-volume-2:20.3.0-5271.g32585015.el9.noarch 82/96 2026-02-17T04:53:19.681 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-volume-2:20.3.0-5271.g32585015.el9.noarch 82/96 2026-02-17T04:53:19.681 INFO:teuthology.orchestra.run.trial100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-17T04:53:19.681 INFO:teuthology.orchestra.run.trial100.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-17T04:53:19.681 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:19.691 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-osd-2:20.3.0-5271.g32585015.el9.x86_64 83/96 2026-02-17T04:53:19.717 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-osd-2:20.3.0-5271.g32585015.el9.x86_64 83/96 2026-02-17T04:53:19.717 INFO:teuthology.orchestra.run.trial100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-17T04:53:19.717 INFO:teuthology.orchestra.run.trial100.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-17T04:53:19.717 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-17T04:53:19.717 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-17T04:53:19.717 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:19.776 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: cephadm-2:20.3.0-5271.g32585015.el9.noarch 84/96 2026-02-17T04:53:19.778 INFO:teuthology.orchestra.run.trial100.stdout: Installing : cephadm-2:20.3.0-5271.g32585015.el9.noarch 84/96 2026-02-17T04:53:19.788 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5271.g32585015.el9 85/96 2026-02-17T04:53:19.812 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5271.g32585015.el 86/96 2026-02-17T04:53:19.813 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5271.g32585015.el9.noarc 87/96 2026-02-17T04:53:22.710 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5271.g32585015.el9.noarc 87/96 2026-02-17T04:53:22.717 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5271.g32585015.el9.noa 88/96 2026-02-17T04:53:23.172 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5271.g32585015.el9.noa 88/96 2026-02-17T04:53:23.173 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5271.g32585 89/96 2026-02-17T04:53:23.188 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5271.g32585 89/96 2026-02-17T04:53:23.189 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5271.g32585015.el9.noa 90/96 2026-02-17T04:53:23.254 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5271.g32585015.el9.noa 90/96 2026-02-17T04:53:23.296 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5271.g32585015.el9. 91/96 2026-02-17T04:53:23.297 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-mgr-2:20.3.0-5271.g32585015.el9.x86_64 92/96 2026-02-17T04:53:23.319 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5271.g32585015.el9.x86_64 92/96 2026-02-17T04:53:23.319 INFO:teuthology.orchestra.run.trial100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-17T04:53:23.319 INFO:teuthology.orchestra.run.trial100.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-17T04:53:23.319 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-17T04:53:23.319 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-17T04:53:23.320 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:23.331 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-mgr-rook-2:20.3.0-5271.g32585015.el9.noarch 93/96 2026-02-17T04:53:23.550 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5271.g32585015.el9.noarch 93/96 2026-02-17T04:53:23.552 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-mon-2:20.3.0-5271.g32585015.el9.x86_64 94/96 2026-02-17T04:53:23.573 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-mon-2:20.3.0-5271.g32585015.el9.x86_64 94/96 2026-02-17T04:53:23.573 INFO:teuthology.orchestra.run.trial100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-17T04:53:23.574 INFO:teuthology.orchestra.run.trial100.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-17T04:53:23.574 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-17T04:53:23.574 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-17T04:53:23.574 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:23.668 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-mds-2:20.3.0-5271.g32585015.el9.x86_64 95/96 2026-02-17T04:53:23.689 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-mds-2:20.3.0-5271.g32585015.el9.x86_64 95/96 2026-02-17T04:53:23.690 INFO:teuthology.orchestra.run.trial100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-17T04:53:23.690 INFO:teuthology.orchestra.run.trial100.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-17T04:53:23.690 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-17T04:53:23.690 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-17T04:53:23.690 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:23.691 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-2:20.3.0-5271.g32585015.el9.x86_64 96/96 2026-02-17T04:53:23.698 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5271.g32585015.el9.x86_6 96/96 2026-02-17T04:53:23.722 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5271.g32585015.el9.x86_6 96/96 2026-02-17T04:53:24.548 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-2:20.3.0-5271.g32585015.el9.x86_64 96/96 2026-02-17T04:53:24.548 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-2:20.3.0-5271.g32585015.el9.x86_64 1/96 2026-02-17T04:53:24.548 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-mds-2:20.3.0-5271.g32585015.el9.x86_64 2/96 2026-02-17T04:53:24.548 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-mgr-2:20.3.0-5271.g32585015.el9.x86_64 3/96 2026-02-17T04:53:24.548 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-mon-2:20.3.0-5271.g32585015.el9.x86_64 4/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-osd-2:20.3.0-5271.g32585015.el9.x86_64 5/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-osd-classic-2:20.3.0-5271.g32585015.el9.x86_6 6/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5271.g32585015.el9.x86_6 7/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5271.g32585015.el 8/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5271.g32585015.el9.noarc 9/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5271.g32585015.el9.noa 10/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5271.g32585 11/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5271.g32585015.el9.noa 12/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5271.g32585015.el9. 13/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5271.g32585015.el9.noarch 14/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5271.g32585015.el9 15/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-volume-2:20.3.0-5271.g32585015.el9.noarch 16/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : cephadm-2:20.3.0-5271.g32585015.el9.noarch 17/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : cryptsetup-2.8.1-2.el9.x86_64 18/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-17T04:53:24.549 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-devel-3.9.25-2.el9.x86_64 40/96 2026-02-17T04:53:24.550 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-17T04:53:24.552 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-17T04:53:24.552 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-17T04:53:24.552 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-17T04:53:24.552 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-17T04:53:24.552 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-17T04:53:24.552 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-17T04:53:24.552 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-17T04:53:24.552 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-17T04:53:24.553 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-17T04:53:24.554 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-17T04:53:24.555 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-17T04:53:24.555 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-17T04:53:24.555 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-17T04:53:24.555 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-17T04:53:24.555 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-17T04:53:24.555 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-17T04:53:24.555 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-17T04:53:24.555 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-17T04:53:24.616 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-17T04:53:24.616 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:24.616 INFO:teuthology.orchestra.run.trial100.stdout:Installed: 2026-02-17T04:53:24.616 INFO:teuthology.orchestra.run.trial100.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-17T04:53:24.616 INFO:teuthology.orchestra.run.trial100.stdout: ceph-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:24.616 INFO:teuthology.orchestra.run.trial100.stdout: ceph-grafana-dashboards-2:20.3.0-5271.g32585015.el9.noarch 2026-02-17T04:53:24.616 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mds-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:24.616 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:24.616 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-cephadm-2:20.3.0-5271.g32585015.el9.noarch 2026-02-17T04:53:24.616 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-dashboard-2:20.3.0-5271.g32585015.el9.noarch 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5271.g32585015.el9.noarch 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-k8sevents-2:20.3.0-5271.g32585015.el9.noarch 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-modules-core-2:20.3.0-5271.g32585015.el9.noarch 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mgr-rook-2:20.3.0-5271.g32585015.el9.noarch 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: ceph-mon-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: ceph-osd-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: ceph-osd-classic-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: ceph-osd-crimson-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: ceph-prometheus-alerts-2:20.3.0-5271.g32585015.el9.noarch 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: ceph-volume-2:20.3.0-5271.g32585015.el9.noarch 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: cephadm-2:20.3.0-5271.g32585015.el9.noarch 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: cryptsetup-2.8.1-2.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: lua-5.4.4-4.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-17T04:53:24.617 INFO:teuthology.orchestra.run.trial100.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-devel-3.9.25-2.el9.x86_64 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-17T04:53:24.618 INFO:teuthology.orchestra.run.trial100.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-ply-3.11-14.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-17T04:53:24.619 INFO:teuthology.orchestra.run.trial100.stdout: unzip-6.0-59.el9.x86_64 2026-02-17T04:53:24.620 INFO:teuthology.orchestra.run.trial100.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-17T04:53:24.620 INFO:teuthology.orchestra.run.trial100.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-17T04:53:24.620 INFO:teuthology.orchestra.run.trial100.stdout: zip-3.0-35.el9.x86_64 2026-02-17T04:53:24.620 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:24.620 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:24.680 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-base 2026-02-17T04:53:24.992 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:36 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:25.056 INFO:teuthology.orchestra.run.trial100.stdout:Package ceph-base-2:20.3.0-5271.g32585015.el9.x86_64 is already installed. 2026-02-17T04:53:25.074 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:25.074 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:25.075 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:25.096 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install cephadm 2026-02-17T04:53:25.406 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:37 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:25.471 INFO:teuthology.orchestra.run.trial100.stdout:Package cephadm-2:20.3.0-5271.g32585015.el9.noarch is already installed. 2026-02-17T04:53:25.488 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:25.489 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:25.489 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:25.512 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-immutable-object-cache 2026-02-17T04:53:25.820 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:37 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:25.901 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout: Package Arch Version Repo Size 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout:Installing: 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5271.g32585015.el9 ceph 147 k 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout:Transaction Summary 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout:Install 1 Package 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout:Total download size: 147 k 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout:Installed size: 419 k 2026-02-17T04:53:25.902 INFO:teuthology.orchestra.run.trial100.stdout:Downloading Packages: 2026-02-17T04:53:26.164 INFO:teuthology.orchestra.run.trial100.stdout:ceph-immutable-object-cache-20.3.0-5271.g325850 563 kB/s | 147 kB 00:00 2026-02-17T04:53:26.164 INFO:teuthology.orchestra.run.trial100.stdout:-------------------------------------------------------------------------------- 2026-02-17T04:53:26.164 INFO:teuthology.orchestra.run.trial100.stdout:Total 559 kB/s | 147 kB 00:00 2026-02-17T04:53:26.165 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction check 2026-02-17T04:53:26.170 INFO:teuthology.orchestra.run.trial100.stdout:Transaction check succeeded. 2026-02-17T04:53:26.170 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction test 2026-02-17T04:53:26.201 INFO:teuthology.orchestra.run.trial100.stdout:Transaction test succeeded. 2026-02-17T04:53:26.201 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction 2026-02-17T04:53:26.255 INFO:teuthology.orchestra.run.trial100.stdout: Preparing : 1/1 2026-02-17T04:53:26.257 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5271.g32585015. 1/1 2026-02-17T04:53:26.280 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5271.g32585015. 1/1 2026-02-17T04:53:26.280 INFO:teuthology.orchestra.run.trial100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-17T04:53:26.280 INFO:teuthology.orchestra.run.trial100.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-17T04:53:26.280 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:26.585 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5271.g32585015. 1/1 2026-02-17T04:53:26.586 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:26.586 INFO:teuthology.orchestra.run.trial100.stdout:Installed: 2026-02-17T04:53:26.586 INFO:teuthology.orchestra.run.trial100.stdout: ceph-immutable-object-cache-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:26.586 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:26.586 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:26.615 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-mgr 2026-02-17T04:53:26.929 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:38 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:26.995 INFO:teuthology.orchestra.run.trial100.stdout:Package ceph-mgr-2:20.3.0-5271.g32585015.el9.x86_64 is already installed. 2026-02-17T04:53:27.013 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:27.014 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:27.014 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:27.034 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-mgr-dashboard 2026-02-17T04:53:27.352 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:39 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:27.416 INFO:teuthology.orchestra.run.trial100.stdout:Package ceph-mgr-dashboard-2:20.3.0-5271.g32585015.el9.noarch is already installed. 2026-02-17T04:53:27.434 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:27.434 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:27.434 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:27.456 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-17T04:53:27.768 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:39 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:27.832 INFO:teuthology.orchestra.run.trial100.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5271.g32585015.el9.noarch is already installed. 2026-02-17T04:53:27.849 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:27.850 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:27.850 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:27.870 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-mgr-rook 2026-02-17T04:53:28.186 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:40 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:28.251 INFO:teuthology.orchestra.run.trial100.stdout:Package ceph-mgr-rook-2:20.3.0-5271.g32585015.el9.noarch is already installed. 2026-02-17T04:53:28.268 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:28.269 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:28.269 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:28.291 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-mgr-cephadm 2026-02-17T04:53:28.597 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:40 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:28.660 INFO:teuthology.orchestra.run.trial100.stdout:Package ceph-mgr-cephadm-2:20.3.0-5271.g32585015.el9.noarch is already installed. 2026-02-17T04:53:28.678 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:28.678 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:28.678 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:28.699 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-osd 2026-02-17T04:53:29.007 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:41 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:29.072 INFO:teuthology.orchestra.run.trial100.stdout:Package ceph-osd-2:20.3.0-5271.g32585015.el9.x86_64 is already installed. 2026-02-17T04:53:29.089 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:29.089 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:29.090 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:29.111 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-osd-classic 2026-02-17T04:53:29.420 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:41 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:29.484 INFO:teuthology.orchestra.run.trial100.stdout:Package ceph-osd-classic-2:20.3.0-5271.g32585015.el9.x86_64 is already installed. 2026-02-17T04:53:29.501 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:29.502 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:29.502 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:29.524 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-fuse 2026-02-17T04:53:29.835 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:41 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:29.918 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:29.918 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:29.918 INFO:teuthology.orchestra.run.trial100.stdout: Package Arch Version Repo Size 2026-02-17T04:53:29.918 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:29.919 INFO:teuthology.orchestra.run.trial100.stdout:Installing: 2026-02-17T04:53:29.919 INFO:teuthology.orchestra.run.trial100.stdout: ceph-fuse x86_64 2:20.3.0-5271.g32585015.el9 ceph 987 k 2026-02-17T04:53:29.919 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:29.919 INFO:teuthology.orchestra.run.trial100.stdout:Transaction Summary 2026-02-17T04:53:29.919 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:29.919 INFO:teuthology.orchestra.run.trial100.stdout:Install 1 Package 2026-02-17T04:53:29.919 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:29.919 INFO:teuthology.orchestra.run.trial100.stdout:Total download size: 987 k 2026-02-17T04:53:29.919 INFO:teuthology.orchestra.run.trial100.stdout:Installed size: 2.9 M 2026-02-17T04:53:29.919 INFO:teuthology.orchestra.run.trial100.stdout:Downloading Packages: 2026-02-17T04:53:30.036 INFO:teuthology.orchestra.run.trial100.stdout:ceph-fuse-20.3.0-5271.g32585015.el9.x86_64.rpm 8.2 MB/s | 987 kB 00:00 2026-02-17T04:53:30.037 INFO:teuthology.orchestra.run.trial100.stdout:-------------------------------------------------------------------------------- 2026-02-17T04:53:30.037 INFO:teuthology.orchestra.run.trial100.stdout:Total 8.1 MB/s | 987 kB 00:00 2026-02-17T04:53:30.037 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction check 2026-02-17T04:53:30.041 INFO:teuthology.orchestra.run.trial100.stdout:Transaction check succeeded. 2026-02-17T04:53:30.041 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction test 2026-02-17T04:53:30.078 INFO:teuthology.orchestra.run.trial100.stdout:Transaction test succeeded. 2026-02-17T04:53:30.078 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction 2026-02-17T04:53:30.171 INFO:teuthology.orchestra.run.trial100.stdout: Preparing : 1/1 2026-02-17T04:53:30.188 INFO:teuthology.orchestra.run.trial100.stdout: Installing : ceph-fuse-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:30.396 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:30.453 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : ceph-fuse-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:30.453 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:30.453 INFO:teuthology.orchestra.run.trial100.stdout:Installed: 2026-02-17T04:53:30.453 INFO:teuthology.orchestra.run.trial100.stdout: ceph-fuse-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:30.453 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:30.453 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:30.481 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install ceph-volume 2026-02-17T04:53:30.798 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:42 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:30.863 INFO:teuthology.orchestra.run.trial100.stdout:Package ceph-volume-2:20.3.0-5271.g32585015.el9.noarch is already installed. 2026-02-17T04:53:30.880 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:30.881 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:30.881 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:30.902 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install librados-devel 2026-02-17T04:53:31.213 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:43 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout: Package Arch Version Repo Size 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout:Installing: 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout: librados-devel x86_64 2:20.3.0-5271.g32585015.el9 ceph 123 k 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout:Transaction Summary 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout:Install 1 Package 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout:Total download size: 123 k 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout:Installed size: 437 k 2026-02-17T04:53:31.295 INFO:teuthology.orchestra.run.trial100.stdout:Downloading Packages: 2026-02-17T04:53:31.388 INFO:teuthology.orchestra.run.trial100.stdout:librados-devel-20.3.0-5271.g32585015.el9.x86_64 1.3 MB/s | 123 kB 00:00 2026-02-17T04:53:31.389 INFO:teuthology.orchestra.run.trial100.stdout:-------------------------------------------------------------------------------- 2026-02-17T04:53:31.389 INFO:teuthology.orchestra.run.trial100.stdout:Total 1.3 MB/s | 123 kB 00:00 2026-02-17T04:53:31.389 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction check 2026-02-17T04:53:31.392 INFO:teuthology.orchestra.run.trial100.stdout:Transaction check succeeded. 2026-02-17T04:53:31.392 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction test 2026-02-17T04:53:31.423 INFO:teuthology.orchestra.run.trial100.stdout:Transaction test succeeded. 2026-02-17T04:53:31.423 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction 2026-02-17T04:53:31.476 INFO:teuthology.orchestra.run.trial100.stdout: Preparing : 1/1 2026-02-17T04:53:31.493 INFO:teuthology.orchestra.run.trial100.stdout: Installing : librados-devel-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:31.574 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: librados-devel-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:31.631 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : librados-devel-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:31.631 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:31.631 INFO:teuthology.orchestra.run.trial100.stdout:Installed: 2026-02-17T04:53:31.631 INFO:teuthology.orchestra.run.trial100.stdout: librados-devel-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:31.631 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:31.631 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:31.659 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install libcephfs2 2026-02-17T04:53:31.980 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:43 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:32.043 INFO:teuthology.orchestra.run.trial100.stdout:Package libcephfs2-2:20.3.0-5271.g32585015.el9.x86_64 is already installed. 2026-02-17T04:53:32.061 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:32.061 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:32.061 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:32.084 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install libcephfs-devel 2026-02-17T04:53:32.391 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:44 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:32.472 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:32.473 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:32.473 INFO:teuthology.orchestra.run.trial100.stdout: Package Arch Version Repo Size 2026-02-17T04:53:32.473 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:32.473 INFO:teuthology.orchestra.run.trial100.stdout:Installing: 2026-02-17T04:53:32.473 INFO:teuthology.orchestra.run.trial100.stdout: libcephfs-devel x86_64 2:20.3.0-5271.g32585015.el9 ceph 37 k 2026-02-17T04:53:32.473 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:32.473 INFO:teuthology.orchestra.run.trial100.stdout:Transaction Summary 2026-02-17T04:53:32.473 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:32.473 INFO:teuthology.orchestra.run.trial100.stdout:Install 1 Package 2026-02-17T04:53:32.473 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:32.474 INFO:teuthology.orchestra.run.trial100.stdout:Total download size: 37 k 2026-02-17T04:53:32.474 INFO:teuthology.orchestra.run.trial100.stdout:Installed size: 171 k 2026-02-17T04:53:32.474 INFO:teuthology.orchestra.run.trial100.stdout:Downloading Packages: 2026-02-17T04:53:32.536 INFO:teuthology.orchestra.run.trial100.stdout:libcephfs-devel-20.3.0-5271.g32585015.el9.x86_6 592 kB/s | 37 kB 00:00 2026-02-17T04:53:32.537 INFO:teuthology.orchestra.run.trial100.stdout:-------------------------------------------------------------------------------- 2026-02-17T04:53:32.537 INFO:teuthology.orchestra.run.trial100.stdout:Total 584 kB/s | 37 kB 00:00 2026-02-17T04:53:32.537 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction check 2026-02-17T04:53:32.539 INFO:teuthology.orchestra.run.trial100.stdout:Transaction check succeeded. 2026-02-17T04:53:32.539 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction test 2026-02-17T04:53:32.541 INFO:teuthology.orchestra.run.trial100.stdout:Transaction test succeeded. 2026-02-17T04:53:32.541 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction 2026-02-17T04:53:32.562 INFO:teuthology.orchestra.run.trial100.stdout: Preparing : 1/1 2026-02-17T04:53:32.579 INFO:teuthology.orchestra.run.trial100.stdout: Installing : libcephfs-devel-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:32.637 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:32.694 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : libcephfs-devel-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:32.694 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:32.694 INFO:teuthology.orchestra.run.trial100.stdout:Installed: 2026-02-17T04:53:32.694 INFO:teuthology.orchestra.run.trial100.stdout: libcephfs-devel-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:32.694 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:32.694 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:32.722 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install librados2 2026-02-17T04:53:33.039 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:45 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:33.105 INFO:teuthology.orchestra.run.trial100.stdout:Package librados2-2:20.3.0-5271.g32585015.el9.x86_64 is already installed. 2026-02-17T04:53:33.123 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:33.123 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:33.123 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:33.146 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install librbd1 2026-02-17T04:53:33.458 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:45 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:33.522 INFO:teuthology.orchestra.run.trial100.stdout:Package librbd1-2:20.3.0-5271.g32585015.el9.x86_64 is already installed. 2026-02-17T04:53:33.539 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:33.540 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:33.540 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:33.561 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install python3-rados 2026-02-17T04:53:33.878 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:45 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:33.941 INFO:teuthology.orchestra.run.trial100.stdout:Package python3-rados-2:20.3.0-5271.g32585015.el9.x86_64 is already installed. 2026-02-17T04:53:33.959 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:33.959 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:33.959 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:33.981 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install python3-rgw 2026-02-17T04:53:34.297 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:46 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:34.361 INFO:teuthology.orchestra.run.trial100.stdout:Package python3-rgw-2:20.3.0-5271.g32585015.el9.x86_64 is already installed. 2026-02-17T04:53:34.379 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:34.379 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:34.380 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:34.402 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install python3-cephfs 2026-02-17T04:53:34.712 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:46 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:34.776 INFO:teuthology.orchestra.run.trial100.stdout:Package python3-cephfs-2:20.3.0-5271.g32585015.el9.x86_64 is already installed. 2026-02-17T04:53:34.793 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:34.794 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:34.794 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:34.816 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install python3-rbd 2026-02-17T04:53:35.138 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:47 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:35.202 INFO:teuthology.orchestra.run.trial100.stdout:Package python3-rbd-2:20.3.0-5271.g32585015.el9.x86_64 is already installed. 2026-02-17T04:53:35.219 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:35.220 INFO:teuthology.orchestra.run.trial100.stdout:Nothing to do. 2026-02-17T04:53:35.220 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:35.242 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install rbd-fuse 2026-02-17T04:53:35.560 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:47 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:35.642 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:35.643 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:35.643 INFO:teuthology.orchestra.run.trial100.stdout: Package Architecture Version Repository Size 2026-02-17T04:53:35.643 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:35.643 INFO:teuthology.orchestra.run.trial100.stdout:Installing: 2026-02-17T04:53:35.643 INFO:teuthology.orchestra.run.trial100.stdout: rbd-fuse x86_64 2:20.3.0-5271.g32585015.el9 ceph 84 k 2026-02-17T04:53:35.643 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:35.643 INFO:teuthology.orchestra.run.trial100.stdout:Transaction Summary 2026-02-17T04:53:35.643 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:35.644 INFO:teuthology.orchestra.run.trial100.stdout:Install 1 Package 2026-02-17T04:53:35.644 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:35.644 INFO:teuthology.orchestra.run.trial100.stdout:Total download size: 84 k 2026-02-17T04:53:35.644 INFO:teuthology.orchestra.run.trial100.stdout:Installed size: 209 k 2026-02-17T04:53:35.644 INFO:teuthology.orchestra.run.trial100.stdout:Downloading Packages: 2026-02-17T04:53:35.717 INFO:teuthology.orchestra.run.trial100.stdout:rbd-fuse-20.3.0-5271.g32585015.el9.x86_64.rpm 1.1 MB/s | 84 kB 00:00 2026-02-17T04:53:35.717 INFO:teuthology.orchestra.run.trial100.stdout:-------------------------------------------------------------------------------- 2026-02-17T04:53:35.717 INFO:teuthology.orchestra.run.trial100.stdout:Total 1.1 MB/s | 84 kB 00:00 2026-02-17T04:53:35.717 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction check 2026-02-17T04:53:35.721 INFO:teuthology.orchestra.run.trial100.stdout:Transaction check succeeded. 2026-02-17T04:53:35.721 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction test 2026-02-17T04:53:35.752 INFO:teuthology.orchestra.run.trial100.stdout:Transaction test succeeded. 2026-02-17T04:53:35.752 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction 2026-02-17T04:53:35.803 INFO:teuthology.orchestra.run.trial100.stdout: Preparing : 1/1 2026-02-17T04:53:35.819 INFO:teuthology.orchestra.run.trial100.stdout: Installing : rbd-fuse-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:35.897 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:35.954 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : rbd-fuse-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:35.954 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:35.954 INFO:teuthology.orchestra.run.trial100.stdout:Installed: 2026-02-17T04:53:35.954 INFO:teuthology.orchestra.run.trial100.stdout: rbd-fuse-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:35.954 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:35.955 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:35.983 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install rbd-mirror 2026-02-17T04:53:36.300 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:48 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:36.381 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout: Package Arch Version Repo Size 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout:Installing: 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout: rbd-mirror x86_64 2:20.3.0-5271.g32585015.el9 ceph 2.9 M 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout:Transaction Summary 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout:Install 1 Package 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout:Total download size: 2.9 M 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout:Installed size: 11 M 2026-02-17T04:53:36.382 INFO:teuthology.orchestra.run.trial100.stdout:Downloading Packages: 2026-02-17T04:53:36.529 INFO:teuthology.orchestra.run.trial100.stdout:rbd-mirror-20.3.0-5271.g32585015.el9.x86_64.rpm 20 MB/s | 2.9 MB 00:00 2026-02-17T04:53:36.529 INFO:teuthology.orchestra.run.trial100.stdout:-------------------------------------------------------------------------------- 2026-02-17T04:53:36.529 INFO:teuthology.orchestra.run.trial100.stdout:Total 19 MB/s | 2.9 MB 00:00 2026-02-17T04:53:36.529 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction check 2026-02-17T04:53:36.534 INFO:teuthology.orchestra.run.trial100.stdout:Transaction check succeeded. 2026-02-17T04:53:36.534 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction test 2026-02-17T04:53:36.573 INFO:teuthology.orchestra.run.trial100.stdout:Transaction test succeeded. 2026-02-17T04:53:36.574 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction 2026-02-17T04:53:36.752 INFO:teuthology.orchestra.run.trial100.stdout: Preparing : 1/1 2026-02-17T04:53:36.754 INFO:teuthology.orchestra.run.trial100.stdout: Installing : rbd-mirror-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:36.777 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:36.777 INFO:teuthology.orchestra.run.trial100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-17T04:53:36.777 INFO:teuthology.orchestra.run.trial100.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-17T04:53:36.777 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-17T04:53:36.778 INFO:teuthology.orchestra.run.trial100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-17T04:53:36.778 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:37.067 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : rbd-mirror-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:37.067 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:37.068 INFO:teuthology.orchestra.run.trial100.stdout:Installed: 2026-02-17T04:53:37.068 INFO:teuthology.orchestra.run.trial100.stdout: rbd-mirror-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:37.068 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:37.068 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:37.098 DEBUG:teuthology.orchestra.run.trial100:> sudo yum -y install rbd-nbd 2026-02-17T04:53:37.412 INFO:teuthology.orchestra.run.trial100.stdout:Last metadata expiration check: 0:00:49 ago on Tue 17 Feb 2026 04:52:48 AM UTC. 2026-02-17T04:53:37.495 INFO:teuthology.orchestra.run.trial100.stdout:Dependencies resolved. 2026-02-17T04:53:37.495 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:37.495 INFO:teuthology.orchestra.run.trial100.stdout: Package Architecture Version Repository Size 2026-02-17T04:53:37.495 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:37.495 INFO:teuthology.orchestra.run.trial100.stdout:Installing: 2026-02-17T04:53:37.495 INFO:teuthology.orchestra.run.trial100.stdout: rbd-nbd x86_64 2:20.3.0-5271.g32585015.el9 ceph 172 k 2026-02-17T04:53:37.496 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:37.496 INFO:teuthology.orchestra.run.trial100.stdout:Transaction Summary 2026-02-17T04:53:37.496 INFO:teuthology.orchestra.run.trial100.stdout:================================================================================ 2026-02-17T04:53:37.496 INFO:teuthology.orchestra.run.trial100.stdout:Install 1 Package 2026-02-17T04:53:37.496 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:37.496 INFO:teuthology.orchestra.run.trial100.stdout:Total download size: 172 k 2026-02-17T04:53:37.496 INFO:teuthology.orchestra.run.trial100.stdout:Installed size: 462 k 2026-02-17T04:53:37.496 INFO:teuthology.orchestra.run.trial100.stdout:Downloading Packages: 2026-02-17T04:53:37.580 INFO:teuthology.orchestra.run.trial100.stdout:rbd-nbd-20.3.0-5271.g32585015.el9.x86_64.rpm 2.0 MB/s | 172 kB 00:00 2026-02-17T04:53:37.580 INFO:teuthology.orchestra.run.trial100.stdout:-------------------------------------------------------------------------------- 2026-02-17T04:53:37.581 INFO:teuthology.orchestra.run.trial100.stdout:Total 2.0 MB/s | 172 kB 00:00 2026-02-17T04:53:37.581 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction check 2026-02-17T04:53:37.585 INFO:teuthology.orchestra.run.trial100.stdout:Transaction check succeeded. 2026-02-17T04:53:37.585 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction test 2026-02-17T04:53:37.616 INFO:teuthology.orchestra.run.trial100.stdout:Transaction test succeeded. 2026-02-17T04:53:37.617 INFO:teuthology.orchestra.run.trial100.stdout:Running transaction 2026-02-17T04:53:37.672 INFO:teuthology.orchestra.run.trial100.stdout: Preparing : 1/1 2026-02-17T04:53:37.688 INFO:teuthology.orchestra.run.trial100.stdout: Installing : rbd-nbd-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:37.764 INFO:teuthology.orchestra.run.trial100.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:37.821 INFO:teuthology.orchestra.run.trial100.stdout: Verifying : rbd-nbd-2:20.3.0-5271.g32585015.el9.x86_64 1/1 2026-02-17T04:53:37.821 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:37.821 INFO:teuthology.orchestra.run.trial100.stdout:Installed: 2026-02-17T04:53:37.821 INFO:teuthology.orchestra.run.trial100.stdout: rbd-nbd-2:20.3.0-5271.g32585015.el9.x86_64 2026-02-17T04:53:37.821 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:37.821 INFO:teuthology.orchestra.run.trial100.stdout:Complete! 2026-02-17T04:53:37.854 DEBUG:teuthology.parallel:result is None 2026-02-17T04:53:37.855 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=325850157d309f65985eea9ec9bb9f1d56726fee 2026-02-17T04:53:38.032 DEBUG:teuthology.orchestra.run.trial100:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-17T04:53:38.054 INFO:teuthology.orchestra.run.trial100.stdout:20.3.0-5271.g32585015.el9 2026-02-17T04:53:38.054 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5271.g32585015.el9 2026-02-17T04:53:38.055 INFO:teuthology.task.install:The correct ceph version 20.3.0-5271.g32585015 is installed. 2026-02-17T04:53:38.058 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-17T04:53:38.059 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:38.059 DEBUG:teuthology.orchestra.run.trial100:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-17T04:53:38.118 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-17T04:53:38.118 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:38.118 DEBUG:teuthology.orchestra.run.trial100:> sudo dd of=/usr/bin/daemon-helper 2026-02-17T04:53:38.185 DEBUG:teuthology.orchestra.run.trial100:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-17T04:53:38.249 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-17T04:53:38.249 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:38.249 DEBUG:teuthology.orchestra.run.trial100:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-17T04:53:38.312 DEBUG:teuthology.orchestra.run.trial100:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-17T04:53:38.375 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-17T04:53:38.375 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:38.375 DEBUG:teuthology.orchestra.run.trial100:> sudo dd of=/usr/bin/stdin-killer 2026-02-17T04:53:38.440 DEBUG:teuthology.orchestra.run.trial100:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-17T04:53:38.503 INFO:teuthology.run_tasks:Running task ceph... 2026-02-17T04:53:38.599 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-17T04:53:38.600 DEBUG:teuthology.orchestra.run.trial100:> sudo chmod 777 /var/log/ceph 2026-02-17T04:53:38.624 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-17T04:53:38.624 DEBUG:teuthology.orchestra.run.trial100:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-17T04:53:38.688 INFO:tasks.ceph:Creating extra log directories... 2026-02-17T04:53:38.688 DEBUG:teuthology.orchestra.run.trial100:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-17T04:53:38.761 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-17T04:53:38.761 INFO:tasks.ceph:config {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '1024K', 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['Error', 'overall HEALTH_', '\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_NEARFULL\\)', '\\(SLOW_OPS\\)', '\\(TOO_FEW_PGS\\)', 'Monitor daemon marked osd\\.[[:digit:]]+ down, but it is still running', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)', '\\(MON_NETSPLIT\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-17T04:53:38.761 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/skanta-2026-02-17_03:53:03-rados-wip-bharath18-testing-2026-02-13-0856-distro-default-trial/50912', 'branch': 'wip-bharath18-testing-2026-02-13-0856', 'description': 'rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/many msgr/async-v2only objectstore/{bluestore/{alloc$/{stupid} base mem$/{normal-1} onode-segment$/{1M} write$/{v2/{compr$/{no$/{no}} v2}}}} rados supported-random-distro$/{centos_latest}}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '50912', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'skanta-2026-02-17_03:53:03-rados-wip-bharath18-testing-2026-02-13-0856-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath18-testing-2026-02-13-0856'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '1024K', 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)', '\\(MON_NETSPLIT\\)'], 'sha1': '325850157d309f65985eea9ec9bb9f1d56726fee'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '325850157d309f65985eea9ec9bb9f1d56726fee'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath18-testing-2026-02-13-0856', 'sha1': '325850157d309f65985eea9ec9bb9f1d56726fee'}}, 'owner': 'scheduled_skanta@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 7109, 'sha1': '325850157d309f65985eea9ec9bb9f1d56726fee', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath18-testing-2026-02-13-0856', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_325850157d309f65985eea9ec9bb9f1d56726fee/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '325850157d309f65985eea9ec9bb9f1d56726fee', 'targets': {'trial100.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOUTuXxdAZDZd8keUQ+KsOPAQTdKnJRgpaS9f2A2d8TgRPHE3RLiXCB/DgWNkwWwODPFxfNIaty3XpzTER+SCE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '1024K', 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'create_rbd_pool': False, 'log-ignorelist': ['Error', 'overall HEALTH_', '\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_NEARFULL\\)', '\\(SLOW_OPS\\)', '\\(TOO_FEW_PGS\\)', 'Monitor daemon marked osd\\.[[:digit:]]+ down, but it is still running', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)', '\\(MON_NETSPLIT\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force'], 'flavor': 'default', 'fs': 'xfs', 'sha1': '325850157d309f65985eea9ec9bb9f1d56726fee', 'cluster': 'ceph'}}, {'backfill_toofull': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-17_03:53:03', 'tube': 'trial', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3682'} 2026-02-17T04:53:38.762 DEBUG:teuthology.orchestra.run.trial100:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-17T04:53:38.809 DEBUG:teuthology.orchestra.run.trial100:> sudo install -d -m0777 -- /var/run/ceph 2026-02-17T04:53:38.876 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:38.876 DEBUG:teuthology.orchestra.run.trial100:> dd if=/scratch_devs of=/dev/stdout 2026-02-17T04:53:38.932 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-17T04:53:38.932 DEBUG:teuthology.orchestra.run.trial100:> stat /dev/vg_nvme/lv_1 2026-02-17T04:53:38.989 INFO:teuthology.orchestra.run.trial100.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-17T04:53:38.989 INFO:teuthology.orchestra.run.trial100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-17T04:53:38.989 INFO:teuthology.orchestra.run.trial100.stdout:Device: 6h/6d Inode: 1309 Links: 1 2026-02-17T04:53:38.989 INFO:teuthology.orchestra.run.trial100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-17T04:53:38.989 INFO:teuthology.orchestra.run.trial100.stdout:Context: system_u:object_r:device_t:s0 2026-02-17T04:53:38.989 INFO:teuthology.orchestra.run.trial100.stdout:Access: 2026-02-17 04:53:36.921401377 +0000 2026-02-17T04:53:38.990 INFO:teuthology.orchestra.run.trial100.stdout:Modify: 2026-02-17 04:52:53.209153926 +0000 2026-02-17T04:53:38.990 INFO:teuthology.orchestra.run.trial100.stdout:Change: 2026-02-17 04:52:53.209153926 +0000 2026-02-17T04:53:38.990 INFO:teuthology.orchestra.run.trial100.stdout: Birth: 2026-02-17 04:52:53.209153926 +0000 2026-02-17T04:53:38.990 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-17T04:53:39.053 INFO:teuthology.orchestra.run.trial100.stderr:1+0 records in 2026-02-17T04:53:39.054 INFO:teuthology.orchestra.run.trial100.stderr:1+0 records out 2026-02-17T04:53:39.054 INFO:teuthology.orchestra.run.trial100.stderr:512 bytes copied, 0.000234675 s, 2.2 MB/s 2026-02-17T04:53:39.055 DEBUG:teuthology.orchestra.run.trial100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-17T04:53:39.112 DEBUG:teuthology.orchestra.run.trial100:> stat /dev/vg_nvme/lv_2 2026-02-17T04:53:39.170 INFO:teuthology.orchestra.run.trial100.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-17T04:53:39.170 INFO:teuthology.orchestra.run.trial100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-17T04:53:39.170 INFO:teuthology.orchestra.run.trial100.stdout:Device: 6h/6d Inode: 1297 Links: 1 2026-02-17T04:53:39.170 INFO:teuthology.orchestra.run.trial100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-17T04:53:39.170 INFO:teuthology.orchestra.run.trial100.stdout:Context: system_u:object_r:device_t:s0 2026-02-17T04:53:39.170 INFO:teuthology.orchestra.run.trial100.stdout:Access: 2026-02-17 04:53:36.921401377 +0000 2026-02-17T04:53:39.170 INFO:teuthology.orchestra.run.trial100.stdout:Modify: 2026-02-17 04:52:53.208153943 +0000 2026-02-17T04:53:39.171 INFO:teuthology.orchestra.run.trial100.stdout:Change: 2026-02-17 04:52:53.208153943 +0000 2026-02-17T04:53:39.171 INFO:teuthology.orchestra.run.trial100.stdout: Birth: 2026-02-17 04:52:53.208153943 +0000 2026-02-17T04:53:39.171 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-17T04:53:39.232 INFO:teuthology.orchestra.run.trial100.stderr:1+0 records in 2026-02-17T04:53:39.233 INFO:teuthology.orchestra.run.trial100.stderr:1+0 records out 2026-02-17T04:53:39.233 INFO:teuthology.orchestra.run.trial100.stderr:512 bytes copied, 0.000213235 s, 2.4 MB/s 2026-02-17T04:53:39.234 DEBUG:teuthology.orchestra.run.trial100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-17T04:53:39.291 DEBUG:teuthology.orchestra.run.trial100:> stat /dev/vg_nvme/lv_3 2026-02-17T04:53:39.347 INFO:teuthology.orchestra.run.trial100.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-17T04:53:39.348 INFO:teuthology.orchestra.run.trial100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-17T04:53:39.348 INFO:teuthology.orchestra.run.trial100.stdout:Device: 6h/6d Inode: 1315 Links: 1 2026-02-17T04:53:39.348 INFO:teuthology.orchestra.run.trial100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-17T04:53:39.348 INFO:teuthology.orchestra.run.trial100.stdout:Context: system_u:object_r:device_t:s0 2026-02-17T04:53:39.348 INFO:teuthology.orchestra.run.trial100.stdout:Access: 2026-02-17 04:53:36.921401377 +0000 2026-02-17T04:53:39.348 INFO:teuthology.orchestra.run.trial100.stdout:Modify: 2026-02-17 04:52:53.209153926 +0000 2026-02-17T04:53:39.348 INFO:teuthology.orchestra.run.trial100.stdout:Change: 2026-02-17 04:52:53.209153926 +0000 2026-02-17T04:53:39.348 INFO:teuthology.orchestra.run.trial100.stdout: Birth: 2026-02-17 04:52:53.209153926 +0000 2026-02-17T04:53:39.349 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-17T04:53:39.410 INFO:teuthology.orchestra.run.trial100.stderr:1+0 records in 2026-02-17T04:53:39.410 INFO:teuthology.orchestra.run.trial100.stderr:1+0 records out 2026-02-17T04:53:39.410 INFO:teuthology.orchestra.run.trial100.stderr:512 bytes copied, 0.000153816 s, 3.3 MB/s 2026-02-17T04:53:39.411 DEBUG:teuthology.orchestra.run.trial100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-17T04:53:39.467 DEBUG:teuthology.orchestra.run.trial100:> stat /dev/vg_nvme/lv_4 2026-02-17T04:53:39.525 INFO:teuthology.orchestra.run.trial100.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-17T04:53:39.525 INFO:teuthology.orchestra.run.trial100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-17T04:53:39.525 INFO:teuthology.orchestra.run.trial100.stdout:Device: 6h/6d Inode: 1293 Links: 1 2026-02-17T04:53:39.525 INFO:teuthology.orchestra.run.trial100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-17T04:53:39.525 INFO:teuthology.orchestra.run.trial100.stdout:Context: system_u:object_r:device_t:s0 2026-02-17T04:53:39.525 INFO:teuthology.orchestra.run.trial100.stdout:Access: 2026-02-17 04:53:36.922401360 +0000 2026-02-17T04:53:39.525 INFO:teuthology.orchestra.run.trial100.stdout:Modify: 2026-02-17 04:52:53.208153943 +0000 2026-02-17T04:53:39.525 INFO:teuthology.orchestra.run.trial100.stdout:Change: 2026-02-17 04:52:53.208153943 +0000 2026-02-17T04:53:39.526 INFO:teuthology.orchestra.run.trial100.stdout: Birth: 2026-02-17 04:52:53.208153943 +0000 2026-02-17T04:53:39.526 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-17T04:53:39.588 INFO:teuthology.orchestra.run.trial100.stderr:1+0 records in 2026-02-17T04:53:39.588 INFO:teuthology.orchestra.run.trial100.stderr:1+0 records out 2026-02-17T04:53:39.588 INFO:teuthology.orchestra.run.trial100.stderr:512 bytes copied, 0.000132957 s, 3.9 MB/s 2026-02-17T04:53:39.589 DEBUG:teuthology.orchestra.run.trial100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-17T04:53:39.647 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-17T04:53:39.647 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial100.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-17T04:53:39.647 INFO:tasks.ceph:Generating config... 2026-02-17T04:53:39.650 INFO:tasks.ceph:[osd] osd max pg log entries = 5 2026-02-17T04:53:39.650 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2026-02-17T04:53:39.650 INFO:tasks.ceph:[osd] bdev async discard = True 2026-02-17T04:53:39.650 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-02-17T04:53:39.650 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2026-02-17T04:53:39.650 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-17T04:53:39.650 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] bluestore onode segment size = 1024K 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] bluestore write v2 = True 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] debug bluefs = 20 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] debug bluestore = 20 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] debug rocksdb = 10 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-17T04:53:39.651 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-17T04:53:39.652 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-17T04:53:39.652 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-17T04:53:39.652 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-17T04:53:39.652 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-17T04:53:39.652 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-17T04:53:39.652 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-17T04:53:39.652 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-17T04:53:39.652 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-17T04:53:39.652 INFO:tasks.ceph:[global] mon client hunt interval max multiple = 2 2026-02-17T04:53:39.652 INFO:tasks.ceph:[global] mon election default strategy = 1 2026-02-17T04:53:39.652 INFO:tasks.ceph:[global] mon mgr beacon grace = 90 2026-02-17T04:53:39.652 INFO:tasks.ceph:[global] ms bind msgr1 = False 2026-02-17T04:53:39.653 INFO:tasks.ceph:[global] ms bind msgr2 = True 2026-02-17T04:53:39.653 INFO:tasks.ceph:[global] ms inject socket failures = 1000 2026-02-17T04:53:39.653 INFO:tasks.ceph:[global] ms type = async 2026-02-17T04:53:39.653 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-17T04:53:39.653 INFO:tasks.ceph:[mgr] debug monc = 10 2026-02-17T04:53:39.653 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-17T04:53:39.653 INFO:tasks.ceph:[mon] debug mon = 30 2026-02-17T04:53:39.653 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-17T04:53:39.653 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-17T04:53:39.653 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-17T04:53:39.653 INFO:tasks.ceph:Setting up mon.a... 2026-02-17T04:53:39.654 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-17T04:53:39.720 INFO:teuthology.orchestra.run.trial100.stdout:creating /etc/ceph/ceph.keyring 2026-02-17T04:53:39.723 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-17T04:53:39.799 DEBUG:teuthology.orchestra.run.trial100:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-17T04:53:39.862 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.100'), ('mon.b', '[v2:10.20.193.100:3301,v1:10.20.193.100:6790]'), ('mon.c', '[v2:10.20.193.100:3302,v1:10.20.193.100:6791]')] 2026-02-17T04:53:39.863 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.100,[v2:10.20.193.100:3301,v1:10.20.193.100:6790],[v2:10.20.193.100:3302,v1:10.20.193.100:6791]', 'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '1024K', 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 30, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2026-02-17T04:53:39.864 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:39.864 DEBUG:teuthology.orchestra.run.trial100:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-17T04:53:39.919 DEBUG:teuthology.orchestra.run.trial100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.100 --addv b '[v2:10.20.193.100:3301,v1:10.20.193.100:6790]' --addv c '[v2:10.20.193.100:3302,v1:10.20.193.100:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2026-02-17T04:53:39.992 INFO:teuthology.orchestra.run.trial100.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-17T04:53:39.993 INFO:teuthology.orchestra.run.trial100.stdout:monmaptool: generated fsid ac0ddb8a-2bd5-421d-9307-ad42ed106d65 2026-02-17T04:53:39.993 INFO:teuthology.orchestra.run.trial100.stdout:setting min_mon_release = tentacle 2026-02-17T04:53:39.993 INFO:teuthology.orchestra.run.trial100.stdout:epoch 0 2026-02-17T04:53:39.993 INFO:teuthology.orchestra.run.trial100.stdout:fsid ac0ddb8a-2bd5-421d-9307-ad42ed106d65 2026-02-17T04:53:39.993 INFO:teuthology.orchestra.run.trial100.stdout:last_changed 2026-02-17T04:53:39.991765+0000 2026-02-17T04:53:39.993 INFO:teuthology.orchestra.run.trial100.stdout:created 2026-02-17T04:53:39.991765+0000 2026-02-17T04:53:39.993 INFO:teuthology.orchestra.run.trial100.stdout:min_mon_release 20 (tentacle) 2026-02-17T04:53:39.993 INFO:teuthology.orchestra.run.trial100.stdout:election_strategy: 1 2026-02-17T04:53:39.993 INFO:teuthology.orchestra.run.trial100.stdout:0: [v2:10.20.193.100:3300/0,v1:10.20.193.100:6789/0] mon.a 2026-02-17T04:53:39.993 INFO:teuthology.orchestra.run.trial100.stdout:1: [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] mon.b 2026-02-17T04:53:39.994 INFO:teuthology.orchestra.run.trial100.stdout:2: [v2:10.20.193.100:3302/0,v1:10.20.193.100:6791/0] mon.c 2026-02-17T04:53:39.994 INFO:teuthology.orchestra.run.trial100.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2026-02-17T04:53:39.995 DEBUG:teuthology.orchestra.run.trial100:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-17T04:53:40.052 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ac0ddb8a-2bd5-421d-9307-ad42ed106d65... 2026-02-17T04:53:40.054 DEBUG:teuthology.orchestra.run.trial100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-17T04:53:40.132 INFO:teuthology.orchestra.run.trial100.stdout:[global] 2026-02-17T04:53:40.132 INFO:teuthology.orchestra.run.trial100.stdout: chdir = "" 2026-02-17T04:53:40.132 INFO:teuthology.orchestra.run.trial100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-17T04:53:40.132 INFO:teuthology.orchestra.run.trial100.stdout: auth supported = cephx 2026-02-17T04:53:40.132 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.132 INFO:teuthology.orchestra.run.trial100.stdout: filestore xattr use omap = true 2026-02-17T04:53:40.132 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.132 INFO:teuthology.orchestra.run.trial100.stdout: mon clock drift allowed = 1.000 2026-02-17T04:53:40.132 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: osd crush chooseleaf type = 0 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: auth debug = true 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: ms die on old message = true 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: ms die on bug = true 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon pg warn max object skew = 0 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: # disable pg_autoscaler by default for new pools 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: osd pool default size = 2 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon osd allow primary affinity = true 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon osd allow pg remap = true 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon warn on legacy crush tunables = false 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon warn on crush straw calc version zero = false 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon warn on no sortbitwise = false 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon warn on osd down out interval zero = false 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon warn on too few osds = false 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon_allow_pool_size_one = true 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: osd default data pool replay window = 5 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.133 INFO:teuthology.orchestra.run.trial100.stdout: mon allow pool delete = true 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: mon cluster log file level = debug 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: debug asserts on shutdown = true 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: mon health detail to clog = false 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: mon host = "10.20.193.100,[v2:10.20.193.100:3301,v1:10.20.193.100:6790],[v2:10.20.193.100:3302,v1:10.20.193.100:6791]" 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: mon client directed command retry = 5 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: mon client hunt interval max multiple = 2 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: mon election default strategy = 1 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: mon mgr beacon grace = 90 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: ms bind msgr1 = False 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: ms bind msgr2 = True 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: ms inject socket failures = 1000 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: ms type = async 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: fsid = ac0ddb8a-2bd5-421d-9307-ad42ed106d65 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout:[osd] 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: osd journal size = 100 2026-02-17T04:53:40.134 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: osd scrub load threshold = 5.0 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: osd scrub max interval = 600 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: osd mclock profile = high_recovery_ops 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: osd mclock skip benchmark = True 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: osd recover clone overlap = true 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: osd recovery max chunk = 1048576 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: osd debug shutdown = true 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: osd debug op order = true 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: osd debug verify stray on activate = true 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: osd debug trim objects = true 2026-02-17T04:53:40.135 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: osd open classes on start = true 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: osd debug pg log writeout = true 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: osd deep scrub update digest min age = 30 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: osd map max advance = 10 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: journal zero on create = true 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: filestore ondisk finisher threads = 3 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: filestore apply finisher threads = 3 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: bdev debug aio = true 2026-02-17T04:53:40.136 INFO:teuthology.orchestra.run.trial100.stdout: osd debug misdirected ops = true 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: osd max pg log entries = 5 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: osd min pg log entries = 5 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: bdev async discard = True 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: bdev enable discard = True 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: bluestore allocator = stupid 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: bluestore block size = 96636764160 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: bluestore fsck on mount = True 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: bluestore onode segment size = 1024K 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: bluestore write v2 = True 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: bluestore zero block detection = True 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: debug bluefs = 20 2026-02-17T04:53:40.137 INFO:teuthology.orchestra.run.trial100.stdout: debug bluestore = 20 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: debug ms = 1 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: debug osd = 20 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: debug rocksdb = 10 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: mon osd backfillfull_ratio = 0.85 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: mon osd full ratio = 0.9 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: mon osd nearfull ratio = 0.8 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: osd debug verify cached snaps = True 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: osd debug verify missing on start = True 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: osd failsafe full ratio = 0.95 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: osd mclock override recovery settings = True 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: osd objectstore = bluestore 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: osd op queue = debug_random 2026-02-17T04:53:40.138 INFO:teuthology.orchestra.run.trial100.stdout: osd op queue cut off = debug_random 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout:[mgr] 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout: debug ms = 1 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout: debug mgr = 20 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout: debug mon = 20 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout: debug auth = 20 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout: mon reweight min pgs per osd = 4 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout: mon reweight min bytes per osd = 10 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout: mgr/telemetry/nag = false 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout: debug monc = 10 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout:[mon] 2026-02-17T04:53:40.139 INFO:teuthology.orchestra.run.trial100.stdout: debug ms = 1 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: debug mon = 30 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: debug paxos = 20 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: debug auth = 20 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: mon data avail warn = 5 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: mon mgr mkfs grace = 240 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: mon reweight min pgs per osd = 4 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: mon osd reporter subtree level = osd 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: mon osd prime pg temp = true 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: mon reweight min bytes per osd = 10 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: auth mon ticket ttl = 660 # 11m 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: auth service ticket ttl = 240 # 4m 2026-02-17T04:53:40.140 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: # don't complain about insecure global_id in the test suite 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: # 1m isn't quite enough 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: mon_down_mkfs_grace = 2m 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: mon_warn_on_filestore_osds = false 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: mon scrub interval = 300 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout:[client] 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: rgw cache enabled = true 2026-02-17T04:53:40.141 INFO:teuthology.orchestra.run.trial100.stdout: rgw enable ops log = true 2026-02-17T04:53:40.142 INFO:teuthology.orchestra.run.trial100.stdout: rgw enable usage log = true 2026-02-17T04:53:40.142 INFO:teuthology.orchestra.run.trial100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-17T04:53:40.142 INFO:teuthology.orchestra.run.trial100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-17T04:53:40.142 INFO:teuthology.orchestra.run.trial100.stdout:[mon.a] 2026-02-17T04:53:40.142 INFO:teuthology.orchestra.run.trial100.stdout:[mon.b] 2026-02-17T04:53:40.142 INFO:teuthology.orchestra.run.trial100.stdout:[mon.c] 2026-02-17T04:53:40.143 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-17T04:53:40.144 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-17T04:53:40.224 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-17T04:53:40.224 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:40.224 DEBUG:teuthology.orchestra.run.trial100:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-17T04:53:40.281 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:40.282 DEBUG:teuthology.orchestra.run.trial100:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-17T04:53:40.339 INFO:tasks.ceph:Sending monmap to node ubuntu@trial100.front.sepia.ceph.com 2026-02-17T04:53:40.339 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:40.339 DEBUG:teuthology.orchestra.run.trial100:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-17T04:53:40.339 DEBUG:teuthology.orchestra.run.trial100:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-17T04:53:40.412 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:40.412 DEBUG:teuthology.orchestra.run.trial100:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-17T04:53:40.468 INFO:tasks.ceph:Setting up mon nodes... 2026-02-17T04:53:40.468 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-17T04:53:40.469 DEBUG:teuthology.orchestra.run.trial100:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-17T04:53:40.551 INFO:teuthology.orchestra.run.trial100.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-17T04:53:40.553 INFO:tasks.ceph:Setting up mds nodes... 2026-02-17T04:53:40.554 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-17T04:53:40.554 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-17T04:53:40.554 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial100.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-17T04:53:40.554 DEBUG:teuthology.orchestra.run.trial100:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-17T04:53:40.617 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-17T04:53:40.617 INFO:tasks.ceph:role: osd.0 2026-02-17T04:53:40.617 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial100.front.sepia.ceph.com 2026-02-17T04:53:40.617 DEBUG:teuthology.orchestra.run.trial100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-17T04:53:40.681 INFO:teuthology.orchestra.run.trial100.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-17T04:53:40.681 INFO:teuthology.orchestra.run.trial100.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-17T04:53:40.681 INFO:teuthology.orchestra.run.trial100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-17T04:53:40.681 INFO:teuthology.orchestra.run.trial100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-17T04:53:40.682 INFO:teuthology.orchestra.run.trial100.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-17T04:53:40.682 INFO:teuthology.orchestra.run.trial100.stdout: = sunit=0 swidth=0 blks 2026-02-17T04:53:40.682 INFO:teuthology.orchestra.run.trial100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-17T04:53:40.682 INFO:teuthology.orchestra.run.trial100.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-17T04:53:40.682 INFO:teuthology.orchestra.run.trial100.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-17T04:53:40.682 INFO:teuthology.orchestra.run.trial100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-17T04:53:41.252 INFO:teuthology.orchestra.run.trial100.stdout:Discarding blocks...Done. 2026-02-17T04:53:41.254 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial100.front.sepia.ceph.com -o noatime 2026-02-17T04:53:41.254 DEBUG:teuthology.orchestra.run.trial100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-17T04:53:41.326 DEBUG:teuthology.orchestra.run.trial100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-17T04:53:41.394 DEBUG:teuthology.orchestra.run.trial100:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-17T04:53:41.459 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-17T04:53:41.459 INFO:tasks.ceph:role: osd.1 2026-02-17T04:53:41.459 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial100.front.sepia.ceph.com 2026-02-17T04:53:41.459 DEBUG:teuthology.orchestra.run.trial100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-17T04:53:41.524 INFO:teuthology.orchestra.run.trial100.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-17T04:53:41.524 INFO:teuthology.orchestra.run.trial100.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-17T04:53:41.524 INFO:teuthology.orchestra.run.trial100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-17T04:53:41.524 INFO:teuthology.orchestra.run.trial100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-17T04:53:41.524 INFO:teuthology.orchestra.run.trial100.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-17T04:53:41.524 INFO:teuthology.orchestra.run.trial100.stdout: = sunit=0 swidth=0 blks 2026-02-17T04:53:41.524 INFO:teuthology.orchestra.run.trial100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-17T04:53:41.524 INFO:teuthology.orchestra.run.trial100.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-17T04:53:41.524 INFO:teuthology.orchestra.run.trial100.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-17T04:53:41.524 INFO:teuthology.orchestra.run.trial100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-17T04:53:42.085 INFO:teuthology.orchestra.run.trial100.stdout:Discarding blocks...Done. 2026-02-17T04:53:42.087 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial100.front.sepia.ceph.com -o noatime 2026-02-17T04:53:42.087 DEBUG:teuthology.orchestra.run.trial100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-17T04:53:42.162 DEBUG:teuthology.orchestra.run.trial100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-17T04:53:42.229 DEBUG:teuthology.orchestra.run.trial100:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-17T04:53:42.294 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-17T04:53:42.294 INFO:tasks.ceph:role: osd.2 2026-02-17T04:53:42.294 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial100.front.sepia.ceph.com 2026-02-17T04:53:42.294 DEBUG:teuthology.orchestra.run.trial100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-17T04:53:42.359 INFO:teuthology.orchestra.run.trial100.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-17T04:53:42.359 INFO:teuthology.orchestra.run.trial100.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-17T04:53:42.359 INFO:teuthology.orchestra.run.trial100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-17T04:53:42.359 INFO:teuthology.orchestra.run.trial100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-17T04:53:42.359 INFO:teuthology.orchestra.run.trial100.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-17T04:53:42.360 INFO:teuthology.orchestra.run.trial100.stdout: = sunit=0 swidth=0 blks 2026-02-17T04:53:42.360 INFO:teuthology.orchestra.run.trial100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-17T04:53:42.360 INFO:teuthology.orchestra.run.trial100.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-17T04:53:42.360 INFO:teuthology.orchestra.run.trial100.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-17T04:53:42.360 INFO:teuthology.orchestra.run.trial100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-17T04:53:42.918 INFO:teuthology.orchestra.run.trial100.stdout:Discarding blocks...Done. 2026-02-17T04:53:42.920 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial100.front.sepia.ceph.com -o noatime 2026-02-17T04:53:42.921 DEBUG:teuthology.orchestra.run.trial100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-17T04:53:42.988 DEBUG:teuthology.orchestra.run.trial100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-17T04:53:43.055 DEBUG:teuthology.orchestra.run.trial100:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-17T04:53:43.118 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-17T04:53:43.118 INFO:tasks.ceph:role: osd.3 2026-02-17T04:53:43.118 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@trial100.front.sepia.ceph.com 2026-02-17T04:53:43.118 DEBUG:teuthology.orchestra.run.trial100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-02-17T04:53:43.183 INFO:teuthology.orchestra.run.trial100.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=45875200 blks 2026-02-17T04:53:43.183 INFO:teuthology.orchestra.run.trial100.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-17T04:53:43.183 INFO:teuthology.orchestra.run.trial100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-17T04:53:43.183 INFO:teuthology.orchestra.run.trial100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-17T04:53:43.183 INFO:teuthology.orchestra.run.trial100.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-17T04:53:43.183 INFO:teuthology.orchestra.run.trial100.stdout: = sunit=0 swidth=0 blks 2026-02-17T04:53:43.183 INFO:teuthology.orchestra.run.trial100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-17T04:53:43.184 INFO:teuthology.orchestra.run.trial100.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-17T04:53:43.184 INFO:teuthology.orchestra.run.trial100.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-17T04:53:43.184 INFO:teuthology.orchestra.run.trial100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-17T04:53:43.740 INFO:teuthology.orchestra.run.trial100.stdout:Discarding blocks...Done. 2026-02-17T04:53:43.742 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@trial100.front.sepia.ceph.com -o noatime 2026-02-17T04:53:43.742 DEBUG:teuthology.orchestra.run.trial100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2026-02-17T04:53:43.825 DEBUG:teuthology.orchestra.run.trial100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-17T04:53:43.891 DEBUG:teuthology.orchestra.run.trial100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-17T04:53:43.968 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:43.967+0000 7f343abfd780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-17T04:53:43.969 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:43.967+0000 7f343abfd780 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-17T04:53:43.969 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:43.967+0000 7f343abfd780 -1 bdev(0x3ca7400 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-17T04:53:43.969 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:43.967+0000 7f343abfd780 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-17T04:53:44.426 DEBUG:teuthology.orchestra.run.trial100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-17T04:53:44.493 DEBUG:teuthology.orchestra.run.trial100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-17T04:53:44.569 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:44.569+0000 7fe5f0068780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-17T04:53:44.570 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:44.569+0000 7fe5f0068780 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-17T04:53:44.571 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:44.569+0000 7fe5f0068780 -1 bdev(0x267b400 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-17T04:53:44.571 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:44.569+0000 7fe5f0068780 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-17T04:53:45.018 DEBUG:teuthology.orchestra.run.trial100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-17T04:53:45.084 DEBUG:teuthology.orchestra.run.trial100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-17T04:53:45.160 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:45.160+0000 7f379f82d780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-17T04:53:45.161 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:45.160+0000 7f379f82d780 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-17T04:53:45.161 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:45.160+0000 7f379f82d780 -1 bdev(0x2ac9400 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-17T04:53:45.161 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:45.160+0000 7f379f82d780 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-17T04:53:45.619 DEBUG:teuthology.orchestra.run.trial100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-17T04:53:45.683 DEBUG:teuthology.orchestra.run.trial100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-17T04:53:45.761 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:45.760+0000 7fb1d7a41780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2026-02-17T04:53:45.761 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:45.760+0000 7fb1d7a41780 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2026-02-17T04:53:45.761 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:45.760+0000 7fb1d7a41780 -1 bdev(0x3259400 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2026-02-17T04:53:45.761 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:53:45.760+0000 7fb1d7a41780 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2026-02-17T04:53:46.216 DEBUG:teuthology.orchestra.run.trial100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-17T04:53:46.283 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-17T04:53:46.283 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:46.283 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-17T04:53:46.347 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:46.347 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-17T04:53:46.411 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:46.411 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-17T04:53:46.476 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:46.476 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-17T04:53:46.539 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:46.540 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-17T04:53:46.604 INFO:tasks.ceph:Adding keys to all mons... 2026-02-17T04:53:46.604 DEBUG:teuthology.orchestra.run.trial100:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-17T04:53:46.666 INFO:teuthology.orchestra.run.trial100.stdout:[mgr.x] 2026-02-17T04:53:46.666 INFO:teuthology.orchestra.run.trial100.stdout: key = AQBU9JNp8zHTIBAAqe5Fg1RzBDazjTYN+xyBeA== 2026-02-17T04:53:46.666 INFO:teuthology.orchestra.run.trial100.stdout:[osd.0] 2026-02-17T04:53:46.666 INFO:teuthology.orchestra.run.trial100.stdout: key = AQBX9JNpsiC4ORAAcZqWDOrJAec7Be9DO6qOGg== 2026-02-17T04:53:46.667 INFO:teuthology.orchestra.run.trial100.stdout:[osd.1] 2026-02-17T04:53:46.667 INFO:teuthology.orchestra.run.trial100.stdout: key = AQBY9JNpWYTvIRAAxiqg41QhUq3BDisqJmpTVg== 2026-02-17T04:53:46.667 INFO:teuthology.orchestra.run.trial100.stdout:[osd.2] 2026-02-17T04:53:46.667 INFO:teuthology.orchestra.run.trial100.stdout: key = AQBZ9JNpUoCSCRAAlDitHSOXvQ2sydmBVcgMMQ== 2026-02-17T04:53:46.667 INFO:teuthology.orchestra.run.trial100.stdout:[osd.3] 2026-02-17T04:53:46.667 INFO:teuthology.orchestra.run.trial100.stdout: key = AQBZ9JNpwNBSLRAAFPIYPaytNOKQBSZJxZSncw== 2026-02-17T04:53:46.668 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-17T04:53:46.748 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-17T04:53:46.827 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-17T04:53:46.905 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-17T04:53:46.982 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-17T04:53:47.060 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-17T04:53:47.061 DEBUG:teuthology.orchestra.run.trial100:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-17T04:53:47.126 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-17T04:53:47.207 DEBUG:teuthology.orchestra.run.trial100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-17T04:53:47.271 DEBUG:teuthology.orchestra.run.trial100:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2026-02-17T04:53:47.336 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-17T04:53:47.417 DEBUG:teuthology.orchestra.run.trial100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2026-02-17T04:53:47.483 DEBUG:teuthology.orchestra.run.trial100:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2026-02-17T04:53:47.547 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-17T04:53:47.628 DEBUG:teuthology.orchestra.run.trial100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2026-02-17T04:53:47.694 DEBUG:teuthology.orchestra.run.trial100:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-17T04:53:47.751 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-17T04:53:47.752 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-17T04:53:47.752 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-17T04:53:47.794 INFO:tasks.ceph.mon.a:Started 2026-02-17T04:53:47.794 INFO:tasks.ceph.mon.b:Restarting daemon 2026-02-17T04:53:47.795 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2026-02-17T04:53:47.797 INFO:tasks.ceph.mon.b:Started 2026-02-17T04:53:47.798 INFO:tasks.ceph.mon.c:Restarting daemon 2026-02-17T04:53:47.798 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2026-02-17T04:53:47.800 INFO:tasks.ceph.mon.c:Started 2026-02-17T04:53:47.801 DEBUG:teuthology.orchestra.run.trial100:> sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force 2026-02-17T04:53:53.914 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-17T04:53:53.914 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-17T04:53:53.914 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-17T04:53:53.916 INFO:tasks.ceph.mgr.x:Started 2026-02-17T04:53:53.916 DEBUG:tasks.ceph:set 0 configs 2026-02-17T04:53:53.916 DEBUG:teuthology.orchestra.run.trial100:> sudo ceph --cluster ceph config dump 2026-02-17T04:53:54.012 INFO:teuthology.orchestra.run.trial100.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-17T04:53:54.012 INFO:teuthology.orchestra.run.trial100.stdout:mgr unknown mgr/devicehealth/enable_monitoring false * 2026-02-17T04:53:54.020 INFO:tasks.ceph:Setting crush tunables to default 2026-02-17T04:53:54.020 DEBUG:teuthology.orchestra.run.trial100:> sudo ceph --cluster ceph osd crush tunables default 2026-02-17T04:53:54.122 INFO:teuthology.orchestra.run.trial100.stderr:adjusted tunables profile to default 2026-02-17T04:53:54.130 INFO:tasks.ceph:check_enable_crimson: False 2026-02-17T04:53:54.130 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-17T04:53:54.130 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:54.130 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-17T04:53:54.156 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:54.156 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-17T04:53:54.218 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:54.218 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-17T04:53:54.280 DEBUG:teuthology.orchestra.run.trial100:> set -ex 2026-02-17T04:53:54.280 DEBUG:teuthology.orchestra.run.trial100:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-17T04:53:54.341 DEBUG:teuthology.orchestra.run.trial100:> sudo ceph --cluster ceph osd new f589f588-f611-4e63-a4a6-8124a0132ffd 0 2026-02-17T04:53:54.483 INFO:teuthology.orchestra.run.trial100.stdout:0 2026-02-17T04:53:54.492 DEBUG:teuthology.orchestra.run.trial100:> sudo ceph --cluster ceph osd new 6d699c04-f7f0-4e53-bbbf-a2cece5b5525 1 2026-02-17T04:53:54.595 INFO:teuthology.orchestra.run.trial100.stdout:1 2026-02-17T04:53:54.603 DEBUG:teuthology.orchestra.run.trial100:> sudo ceph --cluster ceph osd new ff2abd17-d4b9-4c16-8108-10b9a52f0c4d 2 2026-02-17T04:53:54.706 INFO:teuthology.orchestra.run.trial100.stdout:2 2026-02-17T04:53:54.715 DEBUG:teuthology.orchestra.run.trial100:> sudo ceph --cluster ceph osd new 6334acee-00c0-4ca3-8935-a1dfefa95e1d 3 2026-02-17T04:53:54.819 INFO:teuthology.orchestra.run.trial100.stdout:3 2026-02-17T04:53:54.826 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-17T04:53:54.826 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-17T04:53:54.827 INFO:tasks.ceph.osd.0:Started 2026-02-17T04:53:54.828 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-17T04:53:54.828 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-17T04:53:54.829 INFO:tasks.ceph.osd.1:Started 2026-02-17T04:53:54.829 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-17T04:53:54.829 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-17T04:53:54.830 INFO:tasks.ceph.mgr.x.trial100.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-17T04:53:54.830 INFO:tasks.ceph.mgr.x.trial100.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-17T04:53:54.830 INFO:tasks.ceph.mgr.x.trial100.stderr: from numpy import show_config as show_numpy_config 2026-02-17T04:53:54.831 INFO:tasks.ceph.osd.2:Started 2026-02-17T04:53:54.831 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-17T04:53:54.831 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-02-17T04:53:54.832 INFO:tasks.ceph.osd.3:Started 2026-02-17T04:53:54.833 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T04:53:54.933 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:53:54.933 INFO:teuthology.orchestra.run.trial100.stdout:{"epoch":6,"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","created":"2026-02-17T04:53:53.067772+0000","modified":"2026-02-17T04:53:54.817146+0000","last_up_change":"0.000000","last_in_change":"2026-02-17T04:53:54.817146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f589f588-f611-4e63-a4a6-8124a0132ffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d699c04-f7f0-4e53-bbbf-a2cece5b5525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff2abd17-d4b9-4c16-8108-10b9a52f0c4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6334acee-00c0-4ca3-8935-a1dfefa95e1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T04:53:54.940 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-17T04:53:54.940 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-17T04:53:55.110 INFO:tasks.ceph.osd.1.trial100.stderr:2026-02-17T04:53:55.109+0000 7f7d75060780 -1 Falling back to public interface 2026-02-17T04:53:55.118 INFO:tasks.ceph.osd.0.trial100.stderr:2026-02-17T04:53:55.117+0000 7fce9003d780 -1 Falling back to public interface 2026-02-17T04:53:55.119 INFO:tasks.ceph.osd.2.trial100.stderr:2026-02-17T04:53:55.117+0000 7f245a230780 -1 Falling back to public interface 2026-02-17T04:53:55.119 INFO:tasks.ceph.osd.3.trial100.stderr:2026-02-17T04:53:55.117+0000 7fba67859780 -1 Falling back to public interface 2026-02-17T04:53:55.119 INFO:tasks.ceph.osd.2.trial100.stderr:2026-02-17T04:53:55.117+0000 7f24581fd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6804/0: (98) Address already in use 2026-02-17T04:53:55.343 DEBUG:teuthology.orchestra.run.trial100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-17T04:53:55.437 INFO:teuthology.misc.health.trial100.stdout: 2026-02-17T04:53:55.437 INFO:teuthology.misc.health.trial100.stdout:{"epoch":6,"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","created":"2026-02-17T04:53:53.067772+0000","modified":"2026-02-17T04:53:54.817146+0000","last_up_change":"0.000000","last_in_change":"2026-02-17T04:53:54.817146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f589f588-f611-4e63-a4a6-8124a0132ffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d699c04-f7f0-4e53-bbbf-a2cece5b5525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff2abd17-d4b9-4c16-8108-10b9a52f0c4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6334acee-00c0-4ca3-8935-a1dfefa95e1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T04:53:55.444 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-17T04:53:55.784 INFO:tasks.ceph.osd.3.trial100.stderr:2026-02-17T04:53:55.783+0000 7fba67859780 -1 osd.3 0 log_to_monitors true 2026-02-17T04:53:55.800 INFO:tasks.ceph.osd.1.trial100.stderr:2026-02-17T04:53:55.799+0000 7f7d75060780 -1 osd.1 0 log_to_monitors true 2026-02-17T04:53:55.805 INFO:tasks.ceph.osd.0.trial100.stderr:2026-02-17T04:53:55.804+0000 7fce9003d780 -1 osd.0 0 log_to_monitors true 2026-02-17T04:53:55.811 INFO:tasks.ceph.osd.2.trial100.stderr:2026-02-17T04:53:55.810+0000 7f245a230780 -1 osd.2 0 log_to_monitors true 2026-02-17T04:53:57.079 INFO:tasks.ceph.osd.1.trial100.stderr:2026-02-17T04:53:57.078+0000 7f7d703f7640 -1 osd.1 0 waiting for initial osdmap 2026-02-17T04:53:57.079 INFO:tasks.ceph.osd.0.trial100.stderr:2026-02-17T04:53:57.078+0000 7fce8aff7640 -1 osd.0 0 waiting for initial osdmap 2026-02-17T04:53:57.080 INFO:tasks.ceph.osd.2.trial100.stderr:2026-02-17T04:53:57.080+0000 7f24551f7640 -1 osd.2 0 waiting for initial osdmap 2026-02-17T04:53:57.080 INFO:tasks.ceph.osd.3.trial100.stderr:2026-02-17T04:53:57.080+0000 7fba62bf7640 -1 osd.3 0 waiting for initial osdmap 2026-02-17T04:53:57.082 INFO:tasks.ceph.osd.2.trial100.stderr:2026-02-17T04:53:57.081+0000 7f244d1e7640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T04:53:57.082 INFO:tasks.ceph.osd.3.trial100.stderr:2026-02-17T04:53:57.081+0000 7fba5eb32640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T04:53:57.082 INFO:tasks.ceph.osd.0.trial100.stderr:2026-02-17T04:53:57.081+0000 7fce82fe7640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T04:53:57.083 INFO:tasks.ceph.osd.1.trial100.stderr:2026-02-17T04:53:57.081+0000 7f7d683e7640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T04:53:57.263 INFO:tasks.ceph.mgr.x.trial100.stderr:2026-02-17T04:53:57.262+0000 7fb78a6c8640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-02-17T04:53:57.263 INFO:tasks.ceph.mgr.x.trial100.stderr:2026-02-17T04:53:57.262+0000 7fb78a6c8640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-02-17T04:53:57.868 INFO:tasks.ceph.osd.3.trial100.stderr:2026-02-17T04:53:57.867+0000 7fba5963c640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T04:54:01.847 DEBUG:teuthology.orchestra.run.trial100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-17T04:54:02.017 INFO:teuthology.misc.health.trial100.stdout: 2026-02-17T04:54:02.017 INFO:teuthology.misc.health.trial100.stdout:{"epoch":12,"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","created":"2026-02-17T04:53:53.067772+0000","modified":"2026-02-17T04:54:01.263632+0000","last_up_change":"2026-02-17T04:53:58.078956+0000","last_in_change":"2026-02-17T04:53:54.817146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T04:53:58.268259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f589f588-f611-4e63-a4a6-8124a0132ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6805","nonce":119222286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6808","nonce":119222286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6813","nonce":119222286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6811","nonce":119222286}]},"public_addr":"10.20.193.100:6805/119222286","cluster_addr":"10.20.193.100:6808/119222286","heartbeat_back_addr":"10.20.193.100:6813/119222286","heartbeat_front_addr":"10.20.193.100:6811/119222286","state":["exists","up"]},{"osd":1,"uuid":"6d699c04-f7f0-4e53-bbbf-a2cece5b5525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6800","nonce":2214823052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6801","nonce":2214823052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6803","nonce":2214823052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6802","nonce":2214823052}]},"public_addr":"10.20.193.100:6800/2214823052","cluster_addr":"10.20.193.100:6801/2214823052","heartbeat_back_addr":"10.20.193.100:6803/2214823052","heartbeat_front_addr":"10.20.193.100:6802/2214823052","state":["exists","up"]},{"osd":2,"uuid":"ff2abd17-d4b9-4c16-8108-10b9a52f0c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6806","nonce":3399893774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6809","nonce":3399893774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6815","nonce":3399893774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6812","nonce":3399893774}]},"public_addr":"10.20.193.100:6806/3399893774","cluster_addr":"10.20.193.100:6809/3399893774","heartbeat_back_addr":"10.20.193.100:6815/3399893774","heartbeat_front_addr":"10.20.193.100:6812/3399893774","state":["exists","up"]},{"osd":3,"uuid":"6334acee-00c0-4ca3-8935-a1dfefa95e1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6804","nonce":3637649603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6807","nonce":3637649603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6814","nonce":3637649603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6810","nonce":3637649603}]},"public_addr":"10.20.193.100:6804/3637649603","cluster_addr":"10.20.193.100:6807/3637649603","heartbeat_back_addr":"10.20.193.100:6814/3637649603","heartbeat_front_addr":"10.20.193.100:6810/3637649603","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T04:54:02.026 DEBUG:teuthology.misc:4 of 4 OSDs are up 2026-02-17T04:54:02.026 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-17T04:54:02.026 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-17T04:54:02.027 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-17T04:54:02.204 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:02.213 INFO:teuthology.orchestra.run.trial100.stdout:[{"version":2,"timestamp":"2026-02-17T04:53:53.903110+0000","name":"","changes":[{"name":"mgr/mgr/devicehealth/enable_monitoring","new_value":"false"}]}] 2026-02-17T04:54:02.213 INFO:tasks.ceph_manager:config epoch is 2 2026-02-17T04:54:02.213 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-17T04:54:02.214 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-17T04:54:02.214 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-17T04:54:02.422 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:02.432 INFO:teuthology.orchestra.run.trial100.stdout:{"epoch":4,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6816","nonce":729891931}]},"active_addr":"10.20.193.100:6816/729891931","active_change":"2026-02-17T04:53:56.253394+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.100:0","nonce":356295799}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.100:0","nonce":2461791282}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.100:0","nonce":2938746823}]}]} 2026-02-17T04:54:02.439 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-17T04:54:02.439 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-17T04:54:02.440 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T04:54:02.615 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:02.615 INFO:teuthology.orchestra.run.trial100.stdout:{"epoch":12,"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","created":"2026-02-17T04:53:53.067772+0000","modified":"2026-02-17T04:54:01.263632+0000","last_up_change":"2026-02-17T04:53:58.078956+0000","last_in_change":"2026-02-17T04:53:54.817146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T04:53:58.268259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f589f588-f611-4e63-a4a6-8124a0132ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6805","nonce":119222286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6808","nonce":119222286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6813","nonce":119222286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6811","nonce":119222286}]},"public_addr":"10.20.193.100:6805/119222286","cluster_addr":"10.20.193.100:6808/119222286","heartbeat_back_addr":"10.20.193.100:6813/119222286","heartbeat_front_addr":"10.20.193.100:6811/119222286","state":["exists","up"]},{"osd":1,"uuid":"6d699c04-f7f0-4e53-bbbf-a2cece5b5525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6800","nonce":2214823052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6801","nonce":2214823052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6803","nonce":2214823052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6802","nonce":2214823052}]},"public_addr":"10.20.193.100:6800/2214823052","cluster_addr":"10.20.193.100:6801/2214823052","heartbeat_back_addr":"10.20.193.100:6803/2214823052","heartbeat_front_addr":"10.20.193.100:6802/2214823052","state":["exists","up"]},{"osd":2,"uuid":"ff2abd17-d4b9-4c16-8108-10b9a52f0c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6806","nonce":3399893774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6809","nonce":3399893774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6815","nonce":3399893774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6812","nonce":3399893774}]},"public_addr":"10.20.193.100:6806/3399893774","cluster_addr":"10.20.193.100:6809/3399893774","heartbeat_back_addr":"10.20.193.100:6815/3399893774","heartbeat_front_addr":"10.20.193.100:6812/3399893774","state":["exists","up"]},{"osd":3,"uuid":"6334acee-00c0-4ca3-8935-a1dfefa95e1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6804","nonce":3637649603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6807","nonce":3637649603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6814","nonce":3637649603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6810","nonce":3637649603}]},"public_addr":"10.20.193.100:6804/3637649603","cluster_addr":"10.20.193.100:6807/3637649603","heartbeat_back_addr":"10.20.193.100:6814/3637649603","heartbeat_front_addr":"10.20.193.100:6810/3637649603","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T04:54:02.623 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-17T04:54:02.624 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T04:54:02.799 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:02.799 INFO:teuthology.orchestra.run.trial100.stdout:{"epoch":12,"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","created":"2026-02-17T04:53:53.067772+0000","modified":"2026-02-17T04:54:01.263632+0000","last_up_change":"2026-02-17T04:53:58.078956+0000","last_in_change":"2026-02-17T04:53:54.817146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T04:53:58.268259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f589f588-f611-4e63-a4a6-8124a0132ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6805","nonce":119222286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6808","nonce":119222286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6813","nonce":119222286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6811","nonce":119222286}]},"public_addr":"10.20.193.100:6805/119222286","cluster_addr":"10.20.193.100:6808/119222286","heartbeat_back_addr":"10.20.193.100:6813/119222286","heartbeat_front_addr":"10.20.193.100:6811/119222286","state":["exists","up"]},{"osd":1,"uuid":"6d699c04-f7f0-4e53-bbbf-a2cece5b5525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6800","nonce":2214823052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6801","nonce":2214823052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6803","nonce":2214823052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6802","nonce":2214823052}]},"public_addr":"10.20.193.100:6800/2214823052","cluster_addr":"10.20.193.100:6801/2214823052","heartbeat_back_addr":"10.20.193.100:6803/2214823052","heartbeat_front_addr":"10.20.193.100:6802/2214823052","state":["exists","up"]},{"osd":2,"uuid":"ff2abd17-d4b9-4c16-8108-10b9a52f0c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6806","nonce":3399893774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6809","nonce":3399893774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6815","nonce":3399893774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6812","nonce":3399893774}]},"public_addr":"10.20.193.100:6806/3399893774","cluster_addr":"10.20.193.100:6809/3399893774","heartbeat_back_addr":"10.20.193.100:6815/3399893774","heartbeat_front_addr":"10.20.193.100:6812/3399893774","state":["exists","up"]},{"osd":3,"uuid":"6334acee-00c0-4ca3-8935-a1dfefa95e1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6804","nonce":3637649603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6807","nonce":3637649603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6814","nonce":3637649603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6810","nonce":3637649603}]},"public_addr":"10.20.193.100:6804/3637649603","cluster_addr":"10.20.193.100:6807/3637649603","heartbeat_back_addr":"10.20.193.100:6814/3637649603","heartbeat_front_addr":"10.20.193.100:6810/3637649603","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T04:54:02.808 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T04:54:02.809 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T04:54:02.809 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T04:54:02.809 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T04:54:02.896 INFO:teuthology.orchestra.run.trial100.stdout:38654705667 2026-02-17T04:54:02.897 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T04:54:02.898 INFO:teuthology.orchestra.run.trial100.stdout:38654705667 2026-02-17T04:54:02.899 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T04:54:02.900 INFO:teuthology.orchestra.run.trial100.stdout:38654705666 2026-02-17T04:54:02.900 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T04:54:02.901 INFO:teuthology.orchestra.run.trial100.stdout:38654705666 2026-02-17T04:54:02.902 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T04:54:03.118 INFO:teuthology.orchestra.run.trial100.stdout:38654705665 2026-02-17T04:54:03.120 INFO:teuthology.orchestra.run.trial100.stdout:38654705666 2026-02-17T04:54:03.120 INFO:teuthology.orchestra.run.trial100.stdout:38654705666 2026-02-17T04:54:03.122 INFO:teuthology.orchestra.run.trial100.stdout:38654705665 2026-02-17T04:54:03.129 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.1 2026-02-17T04:54:03.132 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.2 2026-02-17T04:54:03.132 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.3 2026-02-17T04:54:03.133 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.0 2026-02-17T04:54:04.130 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T04:54:04.132 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T04:54:04.133 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T04:54:04.135 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T04:54:04.311 INFO:teuthology.orchestra.run.trial100.stdout:38654705668 2026-02-17T04:54:04.314 INFO:teuthology.orchestra.run.trial100.stdout:38654705667 2026-02-17T04:54:04.317 INFO:teuthology.orchestra.run.trial100.stdout:38654705668 2026-02-17T04:54:04.317 INFO:teuthology.orchestra.run.trial100.stdout:38654705667 2026-02-17T04:54:04.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705668 for osd.3 2026-02-17T04:54:04.322 DEBUG:teuthology.parallel:result is None 2026-02-17T04:54:04.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705667 for osd.1 2026-02-17T04:54:04.325 DEBUG:teuthology.parallel:result is None 2026-02-17T04:54:04.328 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705668 for osd.2 2026-02-17T04:54:04.328 DEBUG:teuthology.parallel:result is None 2026-02-17T04:54:04.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705667 for osd.0 2026-02-17T04:54:04.329 DEBUG:teuthology.parallel:result is None 2026-02-17T04:54:04.329 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-17T04:54:04.329 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:54:04.544 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:04.544 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:54:04.553 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-17T04:54:04.257586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":108904,"kb_used_data":1064,"kb_used_omap":22,"kb_used_meta":107753,"kb_avail":377378456,"statfs":{"total":386547056640,"available":386435538944,"internally_reserved":0,"allocated":1089536,"data_stored":757272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23367,"internal_metadata":110339257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"11'17","reported_seq":36,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-17T04:54:01.612947+0000","last_change":"2026-02-17T04:54:00.266150+0000","last_active":"2026-02-17T04:54:01.612947+0000","last_peered":"2026-02-17T04:54:01.612947+0000","last_clean":"2026-02-17T04:54:01.612947+0000","last_became_active":"2026-02-17T04:54:00.266048+0000","last_became_peered":"2026-02-17T04:54:00.266048+0000","last_unstale":"2026-02-17T04:54:01.612947+0000","last_undegraded":"2026-02-17T04:54:01.612947+0000","last_fullsized":"2026-02-17T04:54:01.612947+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T09:09:36.964259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27064,"kb_used_data":104,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":94344776,"statfs":{"total":96636764160,"available":96609050624,"internally_reserved":0,"allocated":106496,"data_stored":25222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":27584490},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27064,"kb_used_data":104,"kb_used_omap":4,"kb_used_meta":26939,"kb_avail":94344776,"statfs":{"total":96636764160,"available":96609050624,"internally_reserved":0,"allocated":106496,"data_stored":25222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4865,"internal_metadata":27585791},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":428,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":438272,"data_stored":353414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":428,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":438272,"data_stored":353414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:54:04.553 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:54:04.727 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:04.728 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:54:04.735 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-17T04:54:04.257586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":108904,"kb_used_data":1064,"kb_used_omap":22,"kb_used_meta":107753,"kb_avail":377378456,"statfs":{"total":386547056640,"available":386435538944,"internally_reserved":0,"allocated":1089536,"data_stored":757272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23367,"internal_metadata":110339257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"11'17","reported_seq":36,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-17T04:54:01.612947+0000","last_change":"2026-02-17T04:54:00.266150+0000","last_active":"2026-02-17T04:54:01.612947+0000","last_peered":"2026-02-17T04:54:01.612947+0000","last_clean":"2026-02-17T04:54:01.612947+0000","last_became_active":"2026-02-17T04:54:00.266048+0000","last_became_peered":"2026-02-17T04:54:00.266048+0000","last_unstale":"2026-02-17T04:54:01.612947+0000","last_undegraded":"2026-02-17T04:54:01.612947+0000","last_fullsized":"2026-02-17T04:54:01.612947+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T09:09:36.964259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27064,"kb_used_data":104,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":94344776,"statfs":{"total":96636764160,"available":96609050624,"internally_reserved":0,"allocated":106496,"data_stored":25222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":27584490},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27064,"kb_used_data":104,"kb_used_omap":4,"kb_used_meta":26939,"kb_avail":94344776,"statfs":{"total":96636764160,"available":96609050624,"internally_reserved":0,"allocated":106496,"data_stored":25222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4865,"internal_metadata":27585791},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":428,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":438272,"data_stored":353414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":428,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":438272,"data_stored":353414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:54:04.736 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-17T04:54:04.736 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-17T04:54:04.736 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-17T04:54:04.737 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-17T04:54:04.935 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:04.935 INFO:teuthology.orchestra.run.trial100.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-17T04:54:04.943 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-17T04:54:04.943 INFO:teuthology.run_tasks:Running task backfill_toofull... 2026-02-17T04:54:04.953 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T04:54:05.130 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:05.130 INFO:teuthology.orchestra.run.trial100.stdout:{"epoch":12,"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","created":"2026-02-17T04:53:53.067772+0000","modified":"2026-02-17T04:54:01.263632+0000","last_up_change":"2026-02-17T04:53:58.078956+0000","last_in_change":"2026-02-17T04:53:54.817146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T04:53:58.268259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f589f588-f611-4e63-a4a6-8124a0132ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6805","nonce":119222286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6808","nonce":119222286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6813","nonce":119222286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6811","nonce":119222286}]},"public_addr":"10.20.193.100:6805/119222286","cluster_addr":"10.20.193.100:6808/119222286","heartbeat_back_addr":"10.20.193.100:6813/119222286","heartbeat_front_addr":"10.20.193.100:6811/119222286","state":["exists","up"]},{"osd":1,"uuid":"6d699c04-f7f0-4e53-bbbf-a2cece5b5525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6800","nonce":2214823052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6801","nonce":2214823052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6803","nonce":2214823052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6802","nonce":2214823052}]},"public_addr":"10.20.193.100:6800/2214823052","cluster_addr":"10.20.193.100:6801/2214823052","heartbeat_back_addr":"10.20.193.100:6803/2214823052","heartbeat_front_addr":"10.20.193.100:6802/2214823052","state":["exists","up"]},{"osd":2,"uuid":"ff2abd17-d4b9-4c16-8108-10b9a52f0c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6806","nonce":3399893774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6809","nonce":3399893774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6815","nonce":3399893774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6812","nonce":3399893774}]},"public_addr":"10.20.193.100:6806/3399893774","cluster_addr":"10.20.193.100:6809/3399893774","heartbeat_back_addr":"10.20.193.100:6815/3399893774","heartbeat_front_addr":"10.20.193.100:6812/3399893774","state":["exists","up"]},{"osd":3,"uuid":"6334acee-00c0-4ca3-8935-a1dfefa95e1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6804","nonce":3637649603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6807","nonce":3637649603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6814","nonce":3637649603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6810","nonce":3637649603}]},"public_addr":"10.20.193.100:6804/3637649603","cluster_addr":"10.20.193.100:6807/3637649603","heartbeat_back_addr":"10.20.193.100:6814/3637649603","heartbeat_front_addr":"10.20.193.100:6810/3637649603","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T04:54:05.139 INFO:tasks.backfill_toofull.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2026-02-17T04:53:58.268259+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'nonprimary_shards': '{}', 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2026-02-17T04:54:05.139 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2026-02-17T04:54:05.314 INFO:teuthology.orchestra.run.trial100.stdout:pg_num: 1 2026-02-17T04:54:05.324 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set backfill_toofull k=2 m=1 crush-failure-domain=osd 2026-02-17T04:54:06.274 INFO:teuthology.orchestra.run.trial100.stderr:load: isa 2026-02-17T04:54:06.286 INFO:tasks.backfill_toofull.ceph_manager:creating pool_name unique_pool_0 2026-02-17T04:54:06.286 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 1 1 erasure backfill_toofull 2026-02-17T04:54:08.278 INFO:teuthology.orchestra.run.trial100.stderr:pool 'unique_pool_0' created 2026-02-17T04:54:08.289 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2026-02-17T04:54:09.283 INFO:teuthology.orchestra.run.trial100.stderr:set pool 2 min_size to 2 2026-02-17T04:54:09.294 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-17T04:54:10.287 INFO:teuthology.orchestra.run.trial100.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-17T04:54:11.298 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_autoscale_mode off 2026-02-17T04:54:12.290 INFO:teuthology.orchestra.run.trial100.stderr:set pool 2 pg_autoscale_mode to off 2026-02-17T04:54:12.304 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T04:54:12.305 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T04:54:12.305 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T04:54:12.306 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T04:54:12.393 INFO:teuthology.orchestra.run.trial100.stdout:38654705670 2026-02-17T04:54:12.393 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T04:54:12.394 INFO:teuthology.orchestra.run.trial100.stdout:38654705669 2026-02-17T04:54:12.395 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T04:54:12.396 INFO:teuthology.orchestra.run.trial100.stdout:38654705670 2026-02-17T04:54:12.396 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T04:54:12.397 INFO:teuthology.orchestra.run.trial100.stdout:38654705669 2026-02-17T04:54:12.398 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T04:54:12.614 INFO:teuthology.orchestra.run.trial100.stdout:38654705668 2026-02-17T04:54:12.617 INFO:teuthology.orchestra.run.trial100.stdout:38654705669 2026-02-17T04:54:12.618 INFO:teuthology.orchestra.run.trial100.stdout:38654705668 2026-02-17T04:54:12.620 INFO:teuthology.orchestra.run.trial100.stdout:38654705669 2026-02-17T04:54:12.626 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.1 2026-02-17T04:54:12.629 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705670 got 38654705669 for osd.3 2026-02-17T04:54:12.630 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.0 2026-02-17T04:54:12.634 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705670 got 38654705669 for osd.2 2026-02-17T04:54:13.627 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T04:54:13.629 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T04:54:13.633 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T04:54:13.635 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T04:54:13.808 INFO:teuthology.orchestra.run.trial100.stdout:38654705669 2026-02-17T04:54:13.810 INFO:teuthology.orchestra.run.trial100.stdout:38654705668 2026-02-17T04:54:13.820 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705670 got 38654705669 for osd.3 2026-02-17T04:54:13.821 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.1 2026-02-17T04:54:13.850 INFO:teuthology.orchestra.run.trial100.stdout:38654705669 2026-02-17T04:54:13.852 INFO:teuthology.orchestra.run.trial100.stdout:38654705668 2026-02-17T04:54:13.861 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705670 got 38654705669 for osd.2 2026-02-17T04:54:13.864 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705668 for osd.0 2026-02-17T04:54:14.821 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T04:54:14.822 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T04:54:14.862 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T04:54:14.865 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T04:54:15.002 INFO:teuthology.orchestra.run.trial100.stdout:38654705670 2026-02-17T04:54:15.005 INFO:teuthology.orchestra.run.trial100.stdout:38654705671 2026-02-17T04:54:15.013 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705670 for osd.1 2026-02-17T04:54:15.014 DEBUG:teuthology.parallel:result is None 2026-02-17T04:54:15.017 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705670 got 38654705671 for osd.3 2026-02-17T04:54:15.018 DEBUG:teuthology.parallel:result is None 2026-02-17T04:54:15.044 INFO:teuthology.orchestra.run.trial100.stdout:38654705671 2026-02-17T04:54:15.048 INFO:teuthology.orchestra.run.trial100.stdout:38654705670 2026-02-17T04:54:15.055 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705670 got 38654705671 for osd.2 2026-02-17T04:54:15.055 DEBUG:teuthology.parallel:result is None 2026-02-17T04:54:15.059 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705669 got 38654705670 for osd.0 2026-02-17T04:54:15.060 DEBUG:teuthology.parallel:result is None 2026-02-17T04:54:15.060 INFO:tasks.backfill_toofull.ceph_manager:waiting for clean 2026-02-17T04:54:15.060 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:54:15.277 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:15.278 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:54:15.286 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2026-02-17T04:54:14.258701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":109096,"kb_used_data":1256,"kb_used_omap":38,"kb_used_meta":107737,"kb_avail":377378264,"statfs":{"total":386547056640,"available":386435342336,"internally_reserved":0,"allocated":1286144,"data_stored":845492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38998,"internal_metadata":110323626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.981295"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-17T04:54:12.291090+0000","last_change":"2026-02-17T04:54:09.308200+0000","last_active":"2026-02-17T04:54:12.291090+0000","last_peered":"2026-02-17T04:54:12.291090+0000","last_clean":"2026-02-17T04:54:12.291090+0000","last_became_active":"2026-02-17T04:54:09.308046+0000","last_became_peered":"2026-02-17T04:54:09.308046+0000","last_unstale":"2026-02-17T04:54:12.291090+0000","last_undegraded":"2026-02-17T04:54:12.291090+0000","last_fullsized":"2026-02-17T04:54:12.291090+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T10:55:27.901276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":48,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-17T04:54:12.290918+0000","last_change":"2026-02-17T04:54:00.266150+0000","last_active":"2026-02-17T04:54:12.290918+0000","last_peered":"2026-02-17T04:54:12.290918+0000","last_clean":"2026-02-17T04:54:12.290918+0000","last_became_active":"2026-02-17T04:54:00.266048+0000","last_became_peered":"2026-02-17T04:54:00.266048+0000","last_unstale":"2026-02-17T04:54:12.290918+0000","last_undegraded":"2026-02-17T04:54:12.290918+0000","last_fullsized":"2026-02-17T04:54:12.290918+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T09:09:36.964259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":152,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":155648,"data_stored":47277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10073,"internal_metadata":27580583},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":152,"kb_used_omap":8,"kb_used_meta":26935,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":155648,"data_stored":47277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8775,"internal_metadata":27581881},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":487424,"data_stored":375469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":27580581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":487424,"data_stored":375469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":27580581},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:54:15.287 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:54:15.461 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:15.461 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:54:15.469 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2026-02-17T04:54:14.258701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":109096,"kb_used_data":1256,"kb_used_omap":38,"kb_used_meta":107737,"kb_avail":377378264,"statfs":{"total":386547056640,"available":386435342336,"internally_reserved":0,"allocated":1286144,"data_stored":845492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38998,"internal_metadata":110323626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.981295"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-17T04:54:12.291090+0000","last_change":"2026-02-17T04:54:09.308200+0000","last_active":"2026-02-17T04:54:12.291090+0000","last_peered":"2026-02-17T04:54:12.291090+0000","last_clean":"2026-02-17T04:54:12.291090+0000","last_became_active":"2026-02-17T04:54:09.308046+0000","last_became_peered":"2026-02-17T04:54:09.308046+0000","last_unstale":"2026-02-17T04:54:12.291090+0000","last_undegraded":"2026-02-17T04:54:12.291090+0000","last_fullsized":"2026-02-17T04:54:12.291090+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T10:55:27.901276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":48,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-17T04:54:12.290918+0000","last_change":"2026-02-17T04:54:00.266150+0000","last_active":"2026-02-17T04:54:12.290918+0000","last_peered":"2026-02-17T04:54:12.290918+0000","last_clean":"2026-02-17T04:54:12.290918+0000","last_became_active":"2026-02-17T04:54:00.266048+0000","last_became_peered":"2026-02-17T04:54:00.266048+0000","last_unstale":"2026-02-17T04:54:12.290918+0000","last_undegraded":"2026-02-17T04:54:12.290918+0000","last_fullsized":"2026-02-17T04:54:12.290918+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T09:09:36.964259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":152,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":155648,"data_stored":47277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10073,"internal_metadata":27580583},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":152,"kb_used_omap":8,"kb_used_meta":26935,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":155648,"data_stored":47277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8775,"internal_metadata":27581881},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":487424,"data_stored":375469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":27580581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":487424,"data_stored":375469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":27580581},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:54:15.470 INFO:tasks.backfill_toofull.ceph_manager:clean! 2026-02-17T04:54:15.471 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T04:54:15.647 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:15.647 INFO:teuthology.orchestra.run.trial100.stdout:{"epoch":18,"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","created":"2026-02-17T04:53:53.067772+0000","modified":"2026-02-17T04:54:12.286378+0000","last_up_change":"2026-02-17T04:53:58.078956+0000","last_in_change":"2026-02-17T04:53:54.817146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T04:53:58.268259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T04:54:07.279692+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_toofull","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"f589f588-f611-4e63-a4a6-8124a0132ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6805","nonce":119222286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6808","nonce":119222286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6813","nonce":119222286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6811","nonce":119222286}]},"public_addr":"10.20.193.100:6805/119222286","cluster_addr":"10.20.193.100:6808/119222286","heartbeat_back_addr":"10.20.193.100:6813/119222286","heartbeat_front_addr":"10.20.193.100:6811/119222286","state":["exists","up"]},{"osd":1,"uuid":"6d699c04-f7f0-4e53-bbbf-a2cece5b5525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6800","nonce":2214823052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6801","nonce":2214823052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6803","nonce":2214823052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6802","nonce":2214823052}]},"public_addr":"10.20.193.100:6800/2214823052","cluster_addr":"10.20.193.100:6801/2214823052","heartbeat_back_addr":"10.20.193.100:6803/2214823052","heartbeat_front_addr":"10.20.193.100:6802/2214823052","state":["exists","up"]},{"osd":2,"uuid":"ff2abd17-d4b9-4c16-8108-10b9a52f0c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6806","nonce":3399893774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6809","nonce":3399893774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6815","nonce":3399893774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6812","nonce":3399893774}]},"public_addr":"10.20.193.100:6806/3399893774","cluster_addr":"10.20.193.100:6809/3399893774","heartbeat_back_addr":"10.20.193.100:6815/3399893774","heartbeat_front_addr":"10.20.193.100:6812/3399893774","state":["exists","up"]},{"osd":3,"uuid":"6334acee-00c0-4ca3-8935-a1dfefa95e1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6804","nonce":3637649603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6807","nonce":3637649603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6814","nonce":3637649603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6810","nonce":3637649603}]},"public_addr":"10.20.193.100:6804/3637649603","cluster_addr":"10.20.193.100:6807/3637649603","heartbeat_back_addr":"10.20.193.100:6814/3637649603","heartbeat_front_addr":"10.20.193.100:6810/3637649603","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T04:53:56.795492+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T04:53:56.802962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T04:53:56.762718+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T04:53:56.823375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_toofull":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T04:54:15.656 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:54:15.828 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:54:15.829 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:54:15.837 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2026-02-17T04:54:14.258701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":109096,"kb_used_data":1256,"kb_used_omap":38,"kb_used_meta":107737,"kb_avail":377378264,"statfs":{"total":386547056640,"available":386435342336,"internally_reserved":0,"allocated":1286144,"data_stored":845492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38998,"internal_metadata":110323626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.981295"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-17T04:54:12.291090+0000","last_change":"2026-02-17T04:54:09.308200+0000","last_active":"2026-02-17T04:54:12.291090+0000","last_peered":"2026-02-17T04:54:12.291090+0000","last_clean":"2026-02-17T04:54:12.291090+0000","last_became_active":"2026-02-17T04:54:09.308046+0000","last_became_peered":"2026-02-17T04:54:09.308046+0000","last_unstale":"2026-02-17T04:54:12.291090+0000","last_undegraded":"2026-02-17T04:54:12.291090+0000","last_fullsized":"2026-02-17T04:54:12.291090+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T10:55:27.901276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":48,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-17T04:54:12.290918+0000","last_change":"2026-02-17T04:54:00.266150+0000","last_active":"2026-02-17T04:54:12.290918+0000","last_peered":"2026-02-17T04:54:12.290918+0000","last_clean":"2026-02-17T04:54:12.290918+0000","last_became_active":"2026-02-17T04:54:00.266048+0000","last_became_peered":"2026-02-17T04:54:00.266048+0000","last_unstale":"2026-02-17T04:54:12.290918+0000","last_undegraded":"2026-02-17T04:54:12.290918+0000","last_fullsized":"2026-02-17T04:54:12.290918+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T09:09:36.964259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":152,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":155648,"data_stored":47277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10073,"internal_metadata":27580583},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":152,"kb_used_omap":8,"kb_used_meta":26935,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":155648,"data_stored":47277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8775,"internal_metadata":27581881},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":487424,"data_stored":375469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":27580581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":487424,"data_stored":375469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":27580581},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:54:15.838 DEBUG:tasks.backfill_toofull:acting=[3, 1, 2] 2026-02-17T04:54:15.838 DEBUG:tasks.backfill_toofull:write some data 2026-02-17T04:54:15.838 INFO:tasks.util.rados:rados -p unique_pool_0 bench 120 write --no-cleanup 2026-02-17T04:54:15.838 DEBUG:teuthology.orchestra.run.trial100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 bench 120 write --no-cleanup 2026-02-17T04:54:15.873 INFO:teuthology.orchestra.run.trial100.stdout:hints = 1 2026-02-17T04:54:15.874 INFO:teuthology.orchestra.run.trial100.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 120 seconds or 0 objects 2026-02-17T04:54:15.874 INFO:teuthology.orchestra.run.trial100.stdout:Object prefix: benchmark_data_trial100_50658 2026-02-17T04:54:15.894 INFO:teuthology.orchestra.run.trial100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T04:54:15.895 INFO:teuthology.orchestra.run.trial100.stdout: 0 0 0 0 0 0 - 0 2026-02-17T04:54:16.138 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:16.136+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:16.365 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:16.363+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4083cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:16.895 INFO:teuthology.orchestra.run.trial100.stdout: 1 16 207 191 763.939 764 0.0526241 0.0814497 2026-02-17T04:54:16.902 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:16.900+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:16.902 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:16.900+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:17.113 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:17.111+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:17.478 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:17.476+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:17.894 INFO:teuthology.orchestra.run.trial100.stdout: 2 16 401 385 769.94 776 0.0498733 0.0820345 2026-02-17T04:54:18.894 INFO:teuthology.orchestra.run.trial100.stdout: 3 16 652 636 847.935 1004 0.0469235 0.0748704 2026-02-17T04:54:19.804 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:19.803+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:19.895 INFO:teuthology.orchestra.run.trial100.stdout: 4 16 909 893 892.932 1028 0.0524403 0.0711503 2026-02-17T04:54:20.423 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:20.421+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:20.435 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:20.433+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:20.727 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:20.726+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:20.894 INFO:teuthology.orchestra.run.trial100.stdout: 5 16 1121 1105 883.934 848 0.0523443 0.0697938 2026-02-17T04:54:21.895 INFO:teuthology.orchestra.run.trial100.stdout: 6 16 1373 1357 904.6 1008 0.02777 0.0705535 2026-02-17T04:54:21.895 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:21.893+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:22.096 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:22.095+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:22.594 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:22.592+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:22.895 INFO:teuthology.orchestra.run.trial100.stdout: 7 16 1500 1484 847.937 508 0.0510528 0.0751979 2026-02-17T04:54:23.095 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:23.093+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab40853f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:23.895 INFO:teuthology.orchestra.run.trial100.stdout: 8 16 1739 1723 861.435 956 0.0526808 0.0740363 2026-02-17T04:54:24.136 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:24.134+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:24.377 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:24.376+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:24.895 INFO:teuthology.orchestra.run.trial100.stdout: 9 16 2058 2042 907.487 1276 0.0445181 0.0703704 2026-02-17T04:54:25.084 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:25.082+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:25.194 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:25.192+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:25.282 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:25.280+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:25.524 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:25.522+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:25.895 INFO:teuthology.orchestra.run.trial100.stdout: 10 16 2364 2348 939.129 1224 0.0559196 0.0679614 2026-02-17T04:54:26.193 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:26.191+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:26.518 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:26.516+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:26.895 INFO:teuthology.orchestra.run.trial100.stdout: 11 16 2683 2667 969.745 1276 0.0523925 0.0658182 2026-02-17T04:54:27.400 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:27.398+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:27.423 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:27.421+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:27.457 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:27.455+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:27.490 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:27.489+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:27.895 INFO:teuthology.orchestra.run.trial100.stdout: 12 16 2933 2917 972.262 1000 0.0553234 0.0656707 2026-02-17T04:54:27.983 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:27.982+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:28.798 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:28.796+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:28.895 INFO:teuthology.orchestra.run.trial100.stdout: 13 16 3193 3177 977.466 1040 0.0532103 0.0653385 2026-02-17T04:54:28.898 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:28.897+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:28.948 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:28.946+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:29.016 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:29.014+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:29.405 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:29.403+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:29.490 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:29.488+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:29.501 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:29.500+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:29.895 INFO:teuthology.orchestra.run.trial100.stdout: 14 16 3447 3431 980.213 1016 0.0430834 0.0651909 2026-02-17T04:54:30.895 INFO:teuthology.orchestra.run.trial100.stdout: 15 16 3533 3517 937.797 344 0.0473304 0.0681296 2026-02-17T04:54:30.901 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:30.900+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:31.046 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:31.044+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:31.896 INFO:teuthology.orchestra.run.trial100.stdout: 16 16 3841 3825 956.179 1232 0.0546491 0.066821 2026-02-17T04:54:32.274 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:32.272+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:32.607 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:32.605+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:32.623 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:32.622+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:32.896 INFO:teuthology.orchestra.run.trial100.stdout: 17 16 4086 4070 957.576 980 0.0512164 0.0667223 2026-02-17T04:54:33.896 INFO:teuthology.orchestra.run.trial100.stdout: 18 16 4371 4355 967.706 1140 0.0450663 0.0656165 2026-02-17T04:54:34.107 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:34.105+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:34.300 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:34.298+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:34.879 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:34.877+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:34.896 INFO:teuthology.orchestra.run.trial100.stdout: 19 16 4662 4646 978.033 1164 0.0526975 0.0653411 2026-02-17T04:54:35.316 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:35.314+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4083880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:35.369 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:35.367+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:35.896 INFO:teuthology.orchestra.run.trial100.stdout:2026-02-17T04:54:35.895786+0000 min lat: 0.00773913 max lat: 0.660277 avg lat: 0.0652157 2026-02-17T04:54:35.896 INFO:teuthology.orchestra.run.trial100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T04:54:35.896 INFO:teuthology.orchestra.run.trial100.stdout: 20 16 4915 4899 979.727 1012 0.0543998 0.0652157 2026-02-17T04:54:36.896 INFO:teuthology.orchestra.run.trial100.stdout: 21 16 5222 5206 991.545 1228 0.0484955 0.0643727 2026-02-17T04:54:37.896 INFO:teuthology.orchestra.run.trial100.stdout: 22 16 5364 5348 972.291 568 0.0530727 0.0657581 2026-02-17T04:54:38.896 INFO:teuthology.orchestra.run.trial100.stdout: 23 16 5677 5661 984.447 1252 0.0509709 0.0649407 2026-02-17T04:54:39.284 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:39.283+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:39.896 INFO:teuthology.orchestra.run.trial100.stdout: 24 16 5874 5858 976.26 788 0.0498855 0.0654828 2026-02-17T04:54:40.896 INFO:teuthology.orchestra.run.trial100.stdout: 25 16 6192 6176 988.086 1272 0.0516072 0.064704 2026-02-17T04:54:41.439 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:41.438+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:41.521 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:41.521+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:41.643 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:41.642+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:41.896 INFO:teuthology.orchestra.run.trial100.stdout: 26 16 6449 6433 989.618 1028 0.0467971 0.0646112 2026-02-17T04:54:42.154 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:42.153+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:42.896 INFO:teuthology.orchestra.run.trial100.stdout: 27 16 6618 6602 978.001 676 0.049389 0.064245 2026-02-17T04:54:43.897 INFO:teuthology.orchestra.run.trial100.stdout: 28 16 6618 6602 943.073 0 - 0.064245 2026-02-17T04:54:44.897 INFO:teuthology.orchestra.run.trial100.stdout: 29 16 6618 6602 910.554 0 - 0.064245 2026-02-17T04:54:45.599 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:45.598+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:45.631 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:45.630+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4085310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:45.897 INFO:teuthology.orchestra.run.trial100.stdout: 30 16 6761 6745 899.268 190.667 0.047738 0.0711115 2026-02-17T04:54:46.117 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:46.117+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:46.136 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:46.136+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:46.191 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:46.190+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:46.896 INFO:teuthology.orchestra.run.trial100.stdout: 31 16 7082 7066 911.675 1284 0.0489724 0.0701382 2026-02-17T04:54:47.715 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:47.714+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:47.816 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:47.815+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:54:47.897 INFO:teuthology.orchestra.run.trial100.stdout: 32 16 7382 7366 920.682 1200 0.0517906 0.069448 2026-02-17T04:54:48.897 INFO:teuthology.orchestra.run.trial100.stdout: 33 16 7699 7683 931.204 1268 0.0538225 0.0686761 2026-02-17T04:54:49.020 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:49.019+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:49.518 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:49.518+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4082d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:49.897 INFO:teuthology.orchestra.run.trial100.stdout: 34 16 8014 7998 940.872 1260 0.0515167 0.0679709 2026-02-17T04:54:50.070 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:50.069+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:50.739 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:50.736+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:50.897 INFO:teuthology.orchestra.run.trial100.stdout: 35 16 8328 8312 949.873 1256 0.0532892 0.067324 2026-02-17T04:54:51.015 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:51.013+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:51.100 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:51.098+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:51.897 INFO:teuthology.orchestra.run.trial100.stdout: 36 16 8607 8591 954.485 1116 0.0481894 0.066837 2026-02-17T04:54:52.376 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:52.374+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:52.804 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:52.802+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:52.813 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:52.811+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:52.897 INFO:teuthology.orchestra.run.trial100.stdout: 37 16 8841 8825 953.983 936 0.045474 0.0670459 2026-02-17T04:54:52.963 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:52.961+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:53.678 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:53.676+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:53.897 INFO:teuthology.orchestra.run.trial100.stdout: 38 16 9091 9075 955.192 1000 0.0444429 0.0669575 2026-02-17T04:54:54.540 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:54.538+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:54.897 INFO:teuthology.orchestra.run.trial100.stdout: 39 16 9364 9348 958.699 1092 0.0395271 0.0665008 2026-02-17T04:54:55.860 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:55.858+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:55.897 INFO:teuthology.orchestra.run.trial100.stdout:2026-02-17T04:54:55.897249+0000 min lat: 0.00734047 max lat: 3.06871 avg lat: 0.0662914 2026-02-17T04:54:55.898 INFO:teuthology.orchestra.run.trial100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T04:54:55.898 INFO:teuthology.orchestra.run.trial100.stdout: 40 16 9664 9648 964.729 1200 0.060476 0.0662914 2026-02-17T04:54:55.952 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:55.950+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:56.149 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:56.147+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:56.897 INFO:teuthology.orchestra.run.trial100.stdout: 41 16 9971 9955 971.148 1228 0.0523702 0.065855 2026-02-17T04:54:57.317 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:57.315+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:57.367 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:57.365+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:57.897 INFO:teuthology.orchestra.run.trial100.stdout: 42 16 10229 10213 972.595 1032 0.0459201 0.0657606 2026-02-17T04:54:58.279 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:54:58.278+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:54:58.897 INFO:teuthology.orchestra.run.trial100.stdout: 43 16 10454 10438 970.905 900 0.0480029 0.0657268 2026-02-17T04:54:59.898 INFO:teuthology.orchestra.run.trial100.stdout: 44 16 10688 10672 970.11 936 0.0496297 0.0659283 2026-02-17T04:55:00.898 INFO:teuthology.orchestra.run.trial100.stdout: 45 16 10999 10983 976.195 1244 0.0492881 0.0655221 2026-02-17T04:55:01.783 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:01.782+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:01.898 INFO:teuthology.orchestra.run.trial100.stdout: 46 16 11228 11212 974.885 916 0.0454399 0.0654497 2026-02-17T04:55:02.017 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:02.015+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:02.898 INFO:teuthology.orchestra.run.trial100.stdout: 47 16 11407 11391 969.376 716 0.054056 0.0657833 2026-02-17T04:55:03.159 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:03.157+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:03.160 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:03.157+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:03.328 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:03.326+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:03.746 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:03.744+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:03.898 INFO:teuthology.orchestra.run.trial100.stdout: 48 16 11649 11633 969.346 968 0.0498485 0.0659891 2026-02-17T04:55:03.945 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:03.943+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:04.195 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:04.193+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:04.287 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:04.285+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:04.898 INFO:teuthology.orchestra.run.trial100.stdout: 49 16 11892 11876 969.398 972 0.0501459 0.0659831 2026-02-17T04:55:05.304 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:05.302+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:05.898 INFO:teuthology.orchestra.run.trial100.stdout: 50 16 12203 12187 974.889 1244 0.0571493 0.0656117 2026-02-17T04:55:06.898 INFO:teuthology.orchestra.run.trial100.stdout: 51 16 12508 12492 979.693 1220 0.0510804 0.0652916 2026-02-17T04:55:07.898 INFO:teuthology.orchestra.run.trial100.stdout: 52 16 12820 12804 984.851 1248 0.0520007 0.0649521 2026-02-17T04:55:08.898 INFO:teuthology.orchestra.run.trial100.stdout: 53 16 13078 13062 985.739 1032 0.0489054 0.0648952 2026-02-17T04:55:08.951 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:08.949+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:09.311 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:09.309+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:09.898 INFO:teuthology.orchestra.run.trial100.stdout: 54 16 13387 13371 990.372 1236 0.0456627 0.0645934 2026-02-17T04:55:10.057 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:10.055+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:10.385 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:10.383+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:10.666 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:10.664+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:10.796 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:10.794+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:10.898 INFO:teuthology.orchestra.run.trial100.stdout: 55 16 13690 13674 994.4 1212 0.0492505 0.0643247 2026-02-17T04:55:11.383 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:11.381+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:11.689 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:11.687+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:11.898 INFO:teuthology.orchestra.run.trial100.stdout: 56 16 13939 13923 994.427 996 0.0434993 0.0643329 2026-02-17T04:55:12.002 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:12.001+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:12.065 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:12.063+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:12.099 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:12.097+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:12.614 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:12.612+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:12.662 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:12.660+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:12.898 INFO:teuthology.orchestra.run.trial100.stdout: 57 16 14249 14233 998.734 1240 0.0497602 0.0640462 2026-02-17T04:55:12.965 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:12.963+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:13.899 INFO:teuthology.orchestra.run.trial100.stdout: 58 16 14367 14351 989.652 472 0.0564889 0.0641782 2026-02-17T04:55:14.486 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:14.484+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:14.548 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:14.547+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:14.899 INFO:teuthology.orchestra.run.trial100.stdout: 59 16 14617 14601 989.826 1000 0.051189 0.0645888 2026-02-17T04:55:15.100 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:15.098+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:15.899 INFO:teuthology.orchestra.run.trial100.stdout:2026-02-17T04:55:15.898665+0000 min lat: 0.00734047 max lat: 3.06871 avg lat: 0.0648622 2026-02-17T04:55:15.899 INFO:teuthology.orchestra.run.trial100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T04:55:15.899 INFO:teuthology.orchestra.run.trial100.stdout: 60 16 14811 14795 986.261 776 0.0538451 0.0648622 2026-02-17T04:55:16.599 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:16.597+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:16.899 INFO:teuthology.orchestra.run.trial100.stdout: 61 16 15058 15042 986.289 988 0.0488958 0.0648567 2026-02-17T04:55:17.899 INFO:teuthology.orchestra.run.trial100.stdout: 62 16 15325 15309 987.605 1068 0.0458183 0.0647723 2026-02-17T04:55:18.198 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:18.196+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:18.899 INFO:teuthology.orchestra.run.trial100.stdout: 63 16 15524 15508 984.563 796 0.0293666 0.0649787 2026-02-17T04:55:18.910 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:18.909+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:19.745 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:19.743+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:19.799 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:19.797+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:19.899 INFO:teuthology.orchestra.run.trial100.stdout: 64 16 15781 15765 985.241 1028 0.0475374 0.0649308 2026-02-17T04:55:20.104 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:20.102+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:20.343 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:20.341+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:20.786 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:20.785+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:20.899 INFO:teuthology.orchestra.run.trial100.stdout: 65 16 16057 16041 987.066 1104 0.0531227 0.0647085 2026-02-17T04:55:21.108 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:21.106+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:21.466 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:21.464+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:21.843 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:21.841+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:21.899 INFO:teuthology.orchestra.run.trial100.stdout: 66 16 16338 16322 989.14 1124 0.0590017 0.0646712 2026-02-17T04:55:22.023 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:22.021+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:22.258 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:22.256+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:22.258 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:22.256+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:22.600 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:22.599+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:22.669 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:22.668+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:22.899 INFO:teuthology.orchestra.run.trial100.stdout: 67 16 16639 16623 992.345 1204 0.0585912 0.0644636 2026-02-17T04:55:23.899 INFO:teuthology.orchestra.run.trial100.stdout: 68 16 16789 16773 986.575 600 0.0427313 0.0646734 2026-02-17T04:55:24.439 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:24.439+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:24.439 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:24.439+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:24.899 INFO:teuthology.orchestra.run.trial100.stdout: 69 16 16984 16968 983.581 780 0.0507721 0.0648848 2026-02-17T04:55:25.241 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:25.240+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:25.426 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:25.425+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:25.438 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:25.437+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:25.900 INFO:teuthology.orchestra.run.trial100.stdout: 70 16 17174 17158 980.386 760 0.0528638 0.065254 2026-02-17T04:55:26.678 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:26.677+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:26.890 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:26.889+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4082d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:26.899 INFO:teuthology.orchestra.run.trial100.stdout: 71 16 17495 17479 984.661 1284 0.0523173 0.0649661 2026-02-17T04:55:27.039 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:27.038+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:27.374 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:27.373+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:27.454 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:27.453+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4082d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:27.552 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:27.551+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4082d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:27.565 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:27.564+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:27.586 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:27.585+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:27.663 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:27.662+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:27.900 INFO:teuthology.orchestra.run.trial100.stdout: 72 16 17796 17780 987.706 1204 0.0492423 0.0647714 2026-02-17T04:55:28.252 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:28.251+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:28.571 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:28.570+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:28.900 INFO:teuthology.orchestra.run.trial100.stdout: 73 16 18106 18090 991.16 1240 0.050905 0.0645444 2026-02-17T04:55:29.900 INFO:teuthology.orchestra.run.trial100.stdout: 74 16 18406 18390 993.982 1200 0.0533279 0.0643605 2026-02-17T04:55:30.371 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:30.370+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:30.533 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:30.532+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:30.871 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:30.870+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:30.900 INFO:teuthology.orchestra.run.trial100.stdout: 75 16 18659 18643 994.221 1012 0.0578908 0.0643471 2026-02-17T04:55:31.277 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:31.276+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:31.277 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:31.276+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:31.900 INFO:teuthology.orchestra.run.trial100.stdout: 76 16 18841 18825 990.717 728 0.253992 0.0644203 2026-02-17T04:55:32.005 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:32.004+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:32.086 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:32.086+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:32.196 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:32.195+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:32.466 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:32.465+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:32.900 INFO:teuthology.orchestra.run.trial100.stdout: 77 16 19095 19079 991.045 1016 0.023221 0.0645696 2026-02-17T04:55:33.900 INFO:teuthology.orchestra.run.trial100.stdout: 78 16 19403 19387 994.133 1232 0.054728 0.0643516 2026-02-17T04:55:34.778 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:34.777+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:34.778 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:34.778+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:34.900 INFO:teuthology.orchestra.run.trial100.stdout: 79 16 19664 19648 994.763 1044 0.051054 0.0643104 2026-02-17T04:55:35.900 INFO:teuthology.orchestra.run.trial100.stdout:2026-02-17T04:55:35.900129+0000 min lat: 0.00734047 max lat: 3.06871 avg lat: 0.0641042 2026-02-17T04:55:35.900 INFO:teuthology.orchestra.run.trial100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T04:55:35.900 INFO:teuthology.orchestra.run.trial100.stdout: 80 16 19977 19961 997.977 1252 0.0532071 0.0641042 2026-02-17T04:55:36.900 INFO:teuthology.orchestra.run.trial100.stdout: 81 16 20237 20221 998.495 1040 0.0497098 0.0640746 2026-02-17T04:55:37.207 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:37.207+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4085150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:37.900 INFO:teuthology.orchestra.run.trial100.stdout: 82 16 20489 20473 998.61 1008 0.0565262 0.064065 2026-02-17T04:55:38.900 INFO:teuthology.orchestra.run.trial100.stdout: 83 16 20741 20725 998.722 1008 0.0485672 0.0640594 2026-02-17T04:55:39.605 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:39.603+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:39.620 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:39.619+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4085150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:39.748 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:39.746+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4085150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:39.749 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:39.747+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:39.900 INFO:teuthology.orchestra.run.trial100.stdout: 84 16 20998 20982 999.07 1028 0.0500273 0.0639216 2026-02-17T04:55:40.006 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:40.004+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:40.555 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:40.553+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:40.848 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:40.846+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:40.901 INFO:teuthology.orchestra.run.trial100.stdout: 85 16 21237 21221 998.562 956 0.0377639 0.064071 2026-02-17T04:55:41.191 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:41.189+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:41.327 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:41.325+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:41.447 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:41.445+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:41.628 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:41.626+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:41.872 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:41.870+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:41.901 INFO:teuthology.orchestra.run.trial100.stdout: 86 16 21536 21520 1000.86 1196 0.0557222 0.0639194 2026-02-17T04:55:42.237 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:42.235+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:42.763 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:42.761+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:42.901 INFO:teuthology.orchestra.run.trial100.stdout: 87 16 21731 21715 998.318 780 0.0476047 0.0640876 2026-02-17T04:55:43.417 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:43.415+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:43.606 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:43.604+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:43.683 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:43.681+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:43.901 INFO:teuthology.orchestra.run.trial100.stdout: 88 16 22032 22016 1000.65 1204 0.044903 0.0639383 2026-02-17T04:55:44.369 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:44.368+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:44.901 INFO:teuthology.orchestra.run.trial100.stdout: 89 16 22344 22328 1003.43 1248 0.0480673 0.0637623 2026-02-17T04:55:45.246 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:45.244+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:45.901 INFO:teuthology.orchestra.run.trial100.stdout: 90 16 22662 22646 1006.42 1272 0.0514552 0.0635714 2026-02-17T04:55:46.481 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:46.479+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:46.901 INFO:teuthology.orchestra.run.trial100.stdout: 91 16 22969 22953 1008.85 1228 0.0566572 0.0634167 2026-02-17T04:55:47.461 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:47.459+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:47.827 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:47.825+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:47.901 INFO:teuthology.orchestra.run.trial100.stdout: 92 16 23194 23178 1007.67 900 0.0488872 0.0634431 2026-02-17T04:55:48.389 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:48.388+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:48.901 INFO:teuthology.orchestra.run.trial100.stdout: 93 16 23412 23396 1006.21 872 0.0489929 0.0635856 2026-02-17T04:55:48.972 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:48.970+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:49.641 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:49.639+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:49.901 INFO:teuthology.orchestra.run.trial100.stdout: 94 16 23717 23701 1008.48 1220 0.0470496 0.0634444 2026-02-17T04:55:50.306 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:50.304+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:50.362 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:50.361+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:50.803 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:50.801+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:50.902 INFO:teuthology.orchestra.run.trial100.stdout: 95 16 23962 23946 1008.18 980 0.0533839 0.063459 2026-02-17T04:55:51.902 INFO:teuthology.orchestra.run.trial100.stdout: 96 16 24091 24075 1003.05 516 0.0177587 0.0637951 2026-02-17T04:55:52.902 INFO:teuthology.orchestra.run.trial100.stdout: 97 16 24410 24394 1005.86 1276 0.0504534 0.0636059 2026-02-17T04:55:53.387 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:53.386+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:53.444 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:53.442+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:53.902 INFO:teuthology.orchestra.run.trial100.stdout: 98 16 24715 24699 1008.05 1220 0.0483578 0.0634703 2026-02-17T04:55:54.013 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:54.011+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:54.902 INFO:teuthology.orchestra.run.trial100.stdout: 99 16 24969 24953 1008.13 1016 0.0540975 0.0634643 2026-02-17T04:55:55.475 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:55.473+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:55.902 INFO:teuthology.orchestra.run.trial100.stdout:2026-02-17T04:55:55.901612+0000 min lat: 0.00734047 max lat: 3.06871 avg lat: 0.0634525 2026-02-17T04:55:55.902 INFO:teuthology.orchestra.run.trial100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T04:55:55.902 INFO:teuthology.orchestra.run.trial100.stdout: 100 16 25226 25210 1008.33 1028 0.0466403 0.0634525 2026-02-17T04:55:56.330 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:56.328+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:56.410 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:56.408+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:56.576 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:56.574+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:56.627 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:56.625+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:56.902 INFO:teuthology.orchestra.run.trial100.stdout: 101 16 25465 25449 1007.81 956 0.0461218 0.0634878 2026-02-17T04:55:57.902 INFO:teuthology.orchestra.run.trial100.stdout: 102 16 25761 25745 1009.53 1184 0.0580771 0.0633759 2026-02-17T04:55:58.089 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:58.087+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:58.172 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:58.170+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:58.336 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:58.334+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:58.676 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:58.674+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:58.757 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:58.755+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:55:58.902 INFO:teuthology.orchestra.run.trial100.stdout: 103 16 26020 26004 1009.79 1036 0.0438708 0.0632755 2026-02-17T04:55:59.413 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:55:59.411+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:55:59.902 INFO:teuthology.orchestra.run.trial100.stdout: 104 16 26255 26239 1009.12 940 0.0530222 0.0634045 2026-02-17T04:56:00.332 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:00.330+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:00.538 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:00.536+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:00.902 INFO:teuthology.orchestra.run.trial100.stdout: 105 16 26446 26430 1006.78 764 0.0216406 0.0635612 2026-02-17T04:56:01.259 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:01.258+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:01.902 INFO:teuthology.orchestra.run.trial100.stdout: 106 16 26702 26686 1006.94 1024 0.0487308 0.0635435 2026-02-17T04:56:01.954 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:01.953+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:02.299 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:02.298+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4084fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:02.903 INFO:teuthology.orchestra.run.trial100.stdout: 107 16 26942 26926 1006.51 960 0.0468123 0.063569 2026-02-17T04:56:03.824 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:03.823+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:03.902 INFO:teuthology.orchestra.run.trial100.stdout: 108 16 27247 27231 1008.48 1220 0.046005 0.0634452 2026-02-17T04:56:04.903 INFO:teuthology.orchestra.run.trial100.stdout: 109 16 27555 27539 1010.53 1232 0.0543967 0.0633148 2026-02-17T04:56:05.903 INFO:teuthology.orchestra.run.trial100.stdout: 110 16 27867 27851 1012.69 1248 0.0484376 0.0631822 2026-02-17T04:56:06.539 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:06.538+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:06.903 INFO:teuthology.orchestra.run.trial100.stdout: 111 16 27985 27969 1007.82 472 0.0474531 0.0634867 2026-02-17T04:56:07.818 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:07.817+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:07.903 INFO:teuthology.orchestra.run.trial100.stdout: 112 16 28255 28239 1008.46 1080 0.0525285 0.0634466 2026-02-17T04:56:08.095 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:08.094+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:08.146 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:08.145+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:08.473 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:08.472+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab40610b0 msgr2=0x7f5ab4081fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:08.589 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:08.588+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:08.903 INFO:teuthology.orchestra.run.trial100.stdout: 113 16 28509 28493 1008.53 1016 0.0571872 0.0633416 2026-02-17T04:56:08.943 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:08.942+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:09.083 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:09.082+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab4081fd0 msgr2=0x7f5ab4084c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:09.185 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:09.184+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:09.274 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:09.273+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab4081fd0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:09.903 INFO:teuthology.orchestra.run.trial100.stdout: 114 16 28627 28611 1003.82 472 0.0539222 0.0634151 2026-02-17T04:56:09.927 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:09.926+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:10.795 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:10.793+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:10.903 INFO:teuthology.orchestra.run.trial100.stdout: 115 16 28877 28861 1003.79 1000 0.0476341 0.0637424 2026-02-17T04:56:11.437 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:11.436+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:11.903 INFO:teuthology.orchestra.run.trial100.stdout: 116 16 29134 29118 1004 1028 0.0534399 0.0637274 2026-02-17T04:56:12.050 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:12.049+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:12.171 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:12.170+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab4081fd0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:12.903 INFO:teuthology.orchestra.run.trial100.stdout: 117 16 29445 29429 1006.05 1244 0.0540265 0.0635985 2026-02-17T04:56:13.498 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:13.497+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:13.700 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:13.698+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab4081fd0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:13.903 INFO:teuthology.orchestra.run.trial100.stdout: 118 16 29691 29675 1005.86 984 0.0459544 0.0635054 2026-02-17T04:56:14.274 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:14.272+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab4081fd0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:14.293 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:14.291+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:14.413 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:14.411+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab4081fd0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:14.456 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:14.454+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab4081fd0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:14.508 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:14.506+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5ab4081fd0 msgr2=0x7f5ab4081490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:14.903 INFO:teuthology.orchestra.run.trial100.stdout: 119 16 30007 29991 1008.03 1264 0.0473582 0.063476 2026-02-17T04:56:15.184 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:15.182+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:15.408 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:15.405+0000 7f5ad3162640 0 -- 10.20.193.100:0/3128831001 >> v2:10.20.193.100:6804/3637649603 conn(0x6ada0f0 msgr2=0x7f5ab4081a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:15.904 INFO:teuthology.orchestra.run.trial100.stdout:2026-02-17T04:56:15.903114+0000 min lat: 0.00734047 max lat: 3.06871 avg lat: 0.063351 2026-02-17T04:56:15.904 INFO:teuthology.orchestra.run.trial100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T04:56:15.904 INFO:teuthology.orchestra.run.trial100.stdout: 120 13 30315 30302 1009.99 1244 0.049982 0.063351 2026-02-17T04:56:16.904 INFO:teuthology.orchestra.run.trial100.stdout:Total time run: 120.036 2026-02-17T04:56:16.904 INFO:teuthology.orchestra.run.trial100.stdout:Total writes made: 30315 2026-02-17T04:56:16.905 INFO:teuthology.orchestra.run.trial100.stdout:Write size: 4194304 2026-02-17T04:56:16.905 INFO:teuthology.orchestra.run.trial100.stdout:Object size: 4194304 2026-02-17T04:56:16.905 INFO:teuthology.orchestra.run.trial100.stdout:Bandwidth (MB/sec): 1010.2 2026-02-17T04:56:16.905 INFO:teuthology.orchestra.run.trial100.stdout:Stddev Bandwidth: 261.175 2026-02-17T04:56:16.905 INFO:teuthology.orchestra.run.trial100.stdout:Max bandwidth (MB/sec): 1284 2026-02-17T04:56:16.905 INFO:teuthology.orchestra.run.trial100.stdout:Min bandwidth (MB/sec): 0 2026-02-17T04:56:16.906 INFO:teuthology.orchestra.run.trial100.stdout:Average IOPS: 252 2026-02-17T04:56:16.906 INFO:teuthology.orchestra.run.trial100.stdout:Stddev IOPS: 65.3112 2026-02-17T04:56:16.906 INFO:teuthology.orchestra.run.trial100.stdout:Max IOPS: 321 2026-02-17T04:56:16.906 INFO:teuthology.orchestra.run.trial100.stdout:Min IOPS: 0 2026-02-17T04:56:16.906 INFO:teuthology.orchestra.run.trial100.stdout:Average Latency(s): 0.0633431 2026-02-17T04:56:16.906 INFO:teuthology.orchestra.run.trial100.stdout:Stddev Latency(s): 0.0879785 2026-02-17T04:56:16.906 INFO:teuthology.orchestra.run.trial100.stdout:Max latency(s): 3.06871 2026-02-17T04:56:16.907 INFO:teuthology.orchestra.run.trial100.stdout:Min latency(s): 0.00734047 2026-02-17T04:56:16.915 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd df name osd.1 --format=json 2026-02-17T04:56:17.134 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:56:17.134 INFO:teuthology.orchestra.run.trial100.stdout:{"nodes":[{"id":1,"device_class":"ssd","name":"osd.1","type":"osd","type_id":0,"crush_weight":0.087890625,"depth":2,"pool_weights":{},"reweight":1,"kb":94371840,"kb_used":60136748,"kb_used_data":59947484,"kb_used_omap":9,"kb_used_meta":189238,"kb_avail":34235092,"utilization":63.72319115532769,"var":1,"pgs":2,"status":"up"}],"stray":[],"summary":{"total_kb":94371840,"total_kb_used":60136748,"total_kb_used_data":59947484,"total_kb_used_omap":9,"total_kb_used_meta":189238,"total_kb_avail":34235092,"average_utilization":63.72319115532769,"min_var":1,"max_var":1,"dev":0}} 2026-02-17T04:56:17.143 DEBUG:tasks.backfill_toofull:target osd df: {'id': 1, 'device_class': 'ssd', 'name': 'osd.1', 'type': 'osd', 'type_id': 0, 'crush_weight': 0.087890625, 'depth': 2, 'pool_weights': {}, 'reweight': 1, 'kb': 94371840, 'kb_used': 60136748, 'kb_used_data': 59947484, 'kb_used_omap': 9, 'kb_used_meta': 189238, 'kb_avail': 34235092, 'utilization': 63.72319115532769, 'var': 1, 'pgs': 2, 'status': 'up'} 2026-02-17T04:56:17.143 DEBUG:tasks.backfill_toofull:pause recovery 2026-02-17T04:56:17.143 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noout 2026-02-17T04:56:17.411 INFO:teuthology.orchestra.run.trial100.stderr:noout is set 2026-02-17T04:56:17.421 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nobackfill 2026-02-17T04:56:18.415 INFO:teuthology.orchestra.run.trial100.stderr:nobackfill is set 2026-02-17T04:56:18.429 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set norecover 2026-02-17T04:56:19.420 INFO:teuthology.orchestra.run.trial100.stderr:norecover is set 2026-02-17T04:56:19.430 DEBUG:tasks.backfill_toofull:stop tartget osd 1 2026-02-17T04:56:19.430 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-17T04:56:19.430 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T04:56:19.478 INFO:tasks.ceph.osd.1:Stopped 2026-02-17T04:56:19.478 INFO:tasks.backfill_toofull.ceph_manager:waiting till active 2026-02-17T04:56:19.478 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:56:19.656 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:56:19.656 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:56:19.664 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2026-02-17T04:56:19.418811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127150653976,"num_objects":30318,"num_object_clones":0,"num_object_copies":90952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30318,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":30353,"num_write_kb":124170563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":186851404,"kb_used_data":186256652,"kb_used_omap":39,"kb_used_meta":594648,"kb_avail":190635956,"statfs":{"total":386547056640,"available":195211218944,"internally_reserved":0,"allocated":190726811648,"data_stored":190726356548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40948,"internal_metadata":608919564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8153727000,"num_objects":1945,"num_object_clones":0,"num_object_copies":5835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1945,"num_write_kb":7962625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.144453"},"pg_stats":[{"pgid":"2.0","version":"18'30316","reported_seq":30336,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-17T04:56:17.410197+0000","last_change":"2026-02-17T04:54:09.308200+0000","last_active":"2026-02-17T04:56:17.410197+0000","last_peered":"2026-02-17T04:56:17.410197+0000","last_clean":"2026-02-17T04:56:17.410197+0000","last_became_active":"2026-02-17T04:54:09.308046+0000","last_became_peered":"2026-02-17T04:54:09.308046+0000","last_unstale":"2026-02-17T04:56:17.410197+0000","last_undegraded":"2026-02-17T04:56:17.410197+0000","last_fullsized":"2026-02-17T04:56:17.410197+0000","mapping_epoch":15,"log_start":"18'30227","ondisk_log_start":"18'30227","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":3000,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T10:55:27.901276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127150325784,"num_objects":30316,"num_object_clones":0,"num_object_copies":90948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30316,"num_write_kb":124170241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-17T04:56:17.410061+0000","last_change":"2026-02-17T04:54:00.266150+0000","last_active":"2026-02-17T04:56:17.410061+0000","last_peered":"2026-02-17T04:56:17.410061+0000","last_clean":"2026-02-17T04:56:17.410061+0000","last_became_active":"2026-02-17T04:54:00.266048+0000","last_became_peered":"2026-02-17T04:54:00.266048+0000","last_unstale":"2026-02-17T04:56:17.410061+0000","last_undegraded":"2026-02-17T04:56:17.410061+0000","last_fullsized":"2026-02-17T04:56:17.410061+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T09:09:36.964259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":127150325784,"num_objects":30316,"num_object_clones":0,"num_object_copies":90948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30316,"num_write_kb":124170241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190725500928,"data_stored":190725500928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705696,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62274548,"kb_used_data":62085284,"kb_used_omap":10,"kb_used_meta":189237,"kb_avail":32097292,"statfs":{"total":96636764160,"available":32867627008,"internally_reserved":0,"allocated":63575330816,"data_stored":63575217629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10723,"internal_metadata":193779229},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62274548,"kb_used_data":62085284,"kb_used_omap":9,"kb_used_meta":189238,"kb_avail":32097292,"statfs":{"total":96636764160,"available":32867627008,"internally_reserved":0,"allocated":63575330816,"data_stored":63575217629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9425,"internal_metadata":193780527},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705695,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62274864,"kb_used_data":62085600,"kb_used_omap":9,"kb_used_meta":189238,"kb_avail":32096976,"statfs":{"total":96636764160,"available":32867303424,"internally_reserved":0,"allocated":63575654400,"data_stored":63575542445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":193779877},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705695,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":484,"kb_used_omap":10,"kb_used_meta":26933,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":495616,"data_stored":378845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10725,"internal_metadata":27579931},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:56:19.666 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:56:19.882 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:56:19.882 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":146,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304038,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":2}],"num_pgs":2,"num_pools":2,"num_objects":29561,"data_bytes":123979760128,"bytes_used":187248463872,"bytes_avail":199298592768,"bytes_total":386547056640,"write_bytes_sec":1049240276,"read_op_per_sec":0,"write_op_per_sec":250},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":2,"modified":"2026-02-17T04:54:58.264859+0000","services":{}},"progress_events":{}} 2026-02-17T04:56:19.891 INFO:tasks.backfill_toofull.ceph_manager:{'fsid': 'ac0ddb8a-2bd5-421d-9307-ad42ed106d65', 'health': {'status': 'HEALTH_WARN', 'checks': {'OSDMAP_FLAGS': {'severity': 'HEALTH_WARN', 'summary': {'message': 'noout,nobackfill,norecover flag(s) set', 'count': 26}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 146, 'monmap': {'epoch': 1, 'min_mon_release_name': 'tentacle', 'num_mons': 3}, 'osdmap': {'epoch': 21, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1771304038, 'num_in_osds': 4, 'osd_in_since': 1771304034, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 2}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 29561, 'data_bytes': 123979760128, 'bytes_used': 187248463872, 'bytes_avail': 199298592768, 'bytes_total': 386547056640, 'write_bytes_sec': 1049240276, 'read_op_per_sec': 0, 'write_op_per_sec': 250}, 'fsmap': {'epoch': 1, 'btime': '2026-02-17T04:53:53:064030+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs'], 'services': {}}, 'servicemap': {'epoch': 2, 'modified': '2026-02-17T04:54:58.264859+0000', 'services': {}}, 'progress_events': {}} 2026-02-17T04:56:19.891 INFO:tasks.backfill_toofull.ceph_manager:active! 2026-02-17T04:56:19.891 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:56:20.064 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:56:20.065 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:56:20.072 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2026-02-17T04:56:19.418811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127150653976,"num_objects":30318,"num_object_clones":0,"num_object_copies":90952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30318,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":30353,"num_write_kb":124170563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":186851404,"kb_used_data":186256652,"kb_used_omap":39,"kb_used_meta":594648,"kb_avail":190635956,"statfs":{"total":386547056640,"available":195211218944,"internally_reserved":0,"allocated":190726811648,"data_stored":190726356548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40948,"internal_metadata":608919564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8153727000,"num_objects":1945,"num_object_clones":0,"num_object_copies":5835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1945,"num_write_kb":7962625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.144453"},"pg_stats":[{"pgid":"2.0","version":"18'30316","reported_seq":30336,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-17T04:56:17.410197+0000","last_change":"2026-02-17T04:54:09.308200+0000","last_active":"2026-02-17T04:56:17.410197+0000","last_peered":"2026-02-17T04:56:17.410197+0000","last_clean":"2026-02-17T04:56:17.410197+0000","last_became_active":"2026-02-17T04:54:09.308046+0000","last_became_peered":"2026-02-17T04:54:09.308046+0000","last_unstale":"2026-02-17T04:56:17.410197+0000","last_undegraded":"2026-02-17T04:56:17.410197+0000","last_fullsized":"2026-02-17T04:56:17.410197+0000","mapping_epoch":15,"log_start":"18'30227","ondisk_log_start":"18'30227","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":3000,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T10:55:27.901276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127150325784,"num_objects":30316,"num_object_clones":0,"num_object_copies":90948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30316,"num_write_kb":124170241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-17T04:56:17.410061+0000","last_change":"2026-02-17T04:54:00.266150+0000","last_active":"2026-02-17T04:56:17.410061+0000","last_peered":"2026-02-17T04:56:17.410061+0000","last_clean":"2026-02-17T04:56:17.410061+0000","last_became_active":"2026-02-17T04:54:00.266048+0000","last_became_peered":"2026-02-17T04:54:00.266048+0000","last_unstale":"2026-02-17T04:56:17.410061+0000","last_undegraded":"2026-02-17T04:56:17.410061+0000","last_fullsized":"2026-02-17T04:56:17.410061+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T09:09:36.964259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":127150325784,"num_objects":30316,"num_object_clones":0,"num_object_copies":90948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30316,"num_write_kb":124170241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190725500928,"data_stored":190725500928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705696,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62274548,"kb_used_data":62085284,"kb_used_omap":10,"kb_used_meta":189237,"kb_avail":32097292,"statfs":{"total":96636764160,"available":32867627008,"internally_reserved":0,"allocated":63575330816,"data_stored":63575217629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10723,"internal_metadata":193779229},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62274548,"kb_used_data":62085284,"kb_used_omap":9,"kb_used_meta":189238,"kb_avail":32097292,"statfs":{"total":96636764160,"available":32867627008,"internally_reserved":0,"allocated":63575330816,"data_stored":63575217629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9425,"internal_metadata":193780527},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705695,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62274864,"kb_used_data":62085600,"kb_used_omap":9,"kb_used_meta":189238,"kb_avail":32096976,"statfs":{"total":96636764160,"available":32867303424,"internally_reserved":0,"allocated":63575654400,"data_stored":63575542445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":193779877},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705695,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":484,"kb_used_omap":10,"kb_used_meta":26933,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":495616,"data_stored":378845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10725,"internal_metadata":27579931},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:56:20.073 DEBUG:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "18'30316", 'reported_seq': 30336, 'reported_epoch': 19, 'state': 'active+clean', 'last_fresh': '2026-02-17T04:56:17.410197+0000', 'last_change': '2026-02-17T04:54:09.308200+0000', 'last_active': '2026-02-17T04:56:17.410197+0000', 'last_peered': '2026-02-17T04:56:17.410197+0000', 'last_clean': '2026-02-17T04:56:17.410197+0000', 'last_became_active': '2026-02-17T04:54:09.308046+0000', 'last_became_peered': '2026-02-17T04:54:09.308046+0000', 'last_unstale': '2026-02-17T04:56:17.410197+0000', 'last_undegraded': '2026-02-17T04:56:17.410197+0000', 'last_fullsized': '2026-02-17T04:56:17.410197+0000', 'mapping_epoch': 15, 'log_start': "18'30227", 'ondisk_log_start': "18'30227", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 89, 'log_dups_size': 3000, 'ondisk_log_size': 89, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-18T10:55:27.901276+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 127150325784, 'num_objects': 30316, 'num_object_clones': 0, 'num_object_copies': 90948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 30316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 30316, 'num_write_kb': 124170241, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:56:20.074 DEBUG:tasks.backfill_toofull:re-write data 2026-02-17T04:56:20.074 INFO:tasks.util.rados:rados -p unique_pool_0 cleanup 2026-02-17T04:56:20.074 DEBUG:teuthology.orchestra.run.trial100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 cleanup 2026-02-17T04:56:21.447 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:21.445+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300610c0 msgr2=0x7f5f300814c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:21.455 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:21.453+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:21.720 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:21.718+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300610c0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:21.720 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:21.718+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:21.921 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:21.919+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:21.961 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:21.959+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300610c0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:22.461 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:22.459+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:22.536 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:22.534+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:22.537 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:22.535+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:22.585 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:22.583+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:22.715 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:22.713+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:22.715 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:22.713+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:22.930 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:22.927+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:22.993 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:22.990+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:23.311 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.309+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:23.346 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.344+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:23.443 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.441+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:23.659 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.657+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:23.672 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.670+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:23.708 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.706+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:23.795 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.793+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:23.838 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.836+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:23.965 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.962+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:23.965 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.963+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:23.969 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.967+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:23.971 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:23.969+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.010 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.008+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.107 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.105+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:24.123 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.121+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.129 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.127+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.143 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.141+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.197 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.195+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.205 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.203+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.205 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.204+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.262 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.260+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.279 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.277+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.445 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.443+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.492 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.490+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.496 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.494+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.681 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.679+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.711 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.709+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.796 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.794+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:24.856 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.854+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:24.864 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:24.862+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:25.273 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:25.271+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:25.561 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:25.559+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:25.576 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:25.574+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:25.591 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:25.589+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:25.859 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:25.857+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:25.928 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:25.926+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:26.137 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:26.135+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:26.202 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:26.199+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:26.204 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:26.202+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:26.211 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:26.209+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:26.450 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:26.447+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:26.466 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:26.463+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:26.622 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:26.620+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:26.700 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:26.698+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:26.771 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:26.769+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:26.850 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:26.849+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:27.091 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:27.088+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:27.102 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:27.100+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:27.261 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:27.259+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:27.292 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:27.290+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:27.295 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:27.294+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:27.317 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:27.314+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:27.384 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:27.382+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:27.491 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:27.489+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:27.639 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:27.637+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:27.851 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:27.849+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:27.878 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:27.875+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:28.221 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:28.218+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:28.227 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:28.225+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:28.330 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:28.328+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:28.681 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:28.679+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:28.685 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:28.683+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:28.783 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:28.780+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:28.783 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:28.781+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:29.005 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:29.003+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:29.033 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:29.031+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:29.063 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:29.061+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:29.074 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:29.072+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:29.086 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:29.084+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:29.613 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:29.611+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:30.309 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:30.307+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:30.362 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:30.360+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:30.482 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:30.480+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:30.527 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:30.525+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:30.569 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:30.567+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:30.591 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:30.589+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:30.677 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:30.675+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:30.898 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:30.896+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:30.943 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:30.941+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:30.953 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:30.951+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:30.959 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:30.957+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.036 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.034+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.048 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.046+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.085 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.083+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.209 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.207+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:31.214 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.212+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.268 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.266+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.456 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.454+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.605 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.603+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.688 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.686+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:31.781 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.779+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.789 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.787+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.830 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.828+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.862 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.860+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.919 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.917+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.937 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.935+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:31.982 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:31.980+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.029 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.027+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.057 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.055+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.087 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.085+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.087 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.085+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.326 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.324+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.335 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.333+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.544 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.542+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.544 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.542+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.785 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.783+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.814 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.812+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.814 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.812+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.957 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.955+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:32.981 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.979+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.001 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:32.999+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.054 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.052+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.071 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.069+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.089 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.087+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.090 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.088+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:33.094 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.092+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.104 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.102+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:33.178 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.176+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:33.227 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.225+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:33.429 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.427+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.455 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.453+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.489 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.487+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:33.556 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.554+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.556 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.554+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:33.805 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.803+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.818 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.816+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.842 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.840+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:33.877 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:33.875+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:34.144 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:34.142+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:34.144 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:34.142+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:34.469 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:34.467+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a25e0 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:34.773 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:34.771+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:34.775 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:34.773+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:34.798 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:34.796+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:34.798 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:34.796+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.011 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.009+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.060 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.058+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.103 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.101+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x18cb020 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.148 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.146+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.166 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.164+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3740 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.189 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.187+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.220 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.217+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:35.254 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.252+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.307 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.305+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:35.328 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.326+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.421 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.419+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.451 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.449+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.465 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.464+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.575 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.573+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:35.718 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.716+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:35.890 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:35.888+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:36.006 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:36.004+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:36.057 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:36.055+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:36.107 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:36.105+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:36.164 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:36.162+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:36.240 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:36.238+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:36.525 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:36.523+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:36.538 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:36.535+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:36.585 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:36.583+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:36.885 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:36.883+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:36.894 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:36.892+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:36.980 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:36.978+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:37.146 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.144+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:37.416 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.414+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:37.426 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.424+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:37.455 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.454+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:37.467 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.465+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:37.708 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.706+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:37.722 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.720+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:37.743 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.741+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:37.790 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.788+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:37.803 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.802+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:37.845 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.843+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:37.987 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:37.985+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:38.189 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:38.187+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:38.260 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:38.258+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:38.340 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:38.338+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:38.366 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:38.364+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:38.790 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:38.789+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:39.125 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.123+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:39.305 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.303+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:39.336 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.334+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:39.342 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.340+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:39.594 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.592+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:39.616 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.614+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:39.641 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.639+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:39.667 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.665+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:39.695 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.693+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:39.706 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.704+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3b10 msgr2=0x7f5f300a20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:39.714 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.712+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:39.825 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:39.823+0000 7f5f43fff640 0 -- 10.20.193.100:0/3767174770 >> v2:10.20.193.100:6804/3637649603 conn(0x7f5f300a3370 msgr2=0x7f5f300814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:39.874 INFO:teuthology.orchestra.run.trial100.stdout:Removed 30315 objects 2026-02-17T04:56:49.878 INFO:tasks.util.rados:rados -p unique_pool_0 bench 60 write --no-cleanup 2026-02-17T04:56:49.879 DEBUG:teuthology.orchestra.run.trial100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 bench 60 write --no-cleanup 2026-02-17T04:56:49.916 INFO:teuthology.orchestra.run.trial100.stdout:hints = 1 2026-02-17T04:56:49.917 INFO:teuthology.orchestra.run.trial100.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 60 seconds or 0 objects 2026-02-17T04:56:49.917 INFO:teuthology.orchestra.run.trial100.stdout:Object prefix: benchmark_data_trial100_51457 2026-02-17T04:56:49.936 INFO:teuthology.orchestra.run.trial100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T04:56:49.936 INFO:teuthology.orchestra.run.trial100.stdout: 0 0 0 0 0 0 - 0 2026-02-17T04:56:50.155 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:50.154+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x6616d00 msgr2=0x7f6e84081b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:50.192 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:50.191+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x6616d00 msgr2=0x7f6e840850b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:50.487 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:50.485+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:50.487 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:50.485+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:50.497 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:50.496+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x6616d00 msgr2=0x7f6e84081af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:50.649 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:50.648+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x6616d00 msgr2=0x7f6e84083e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:50.936 INFO:teuthology.orchestra.run.trial100.stdout: 1 16 385 369 1475.89 1476 0.0385044 0.042586 2026-02-17T04:56:51.936 INFO:teuthology.orchestra.run.trial100.stdout: 2 16 759 743 1485.89 1496 0.0444387 0.0425998 2026-02-17T04:56:52.220 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:52.219+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x6616d00 msgr2=0x7f6e84084270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:52.564 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:52.562+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x6616d00 msgr2=0x7f6e84084df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:52.773 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:52.772+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:52.936 INFO:teuthology.orchestra.run.trial100.stdout: 3 16 1059 1043 1390.57 1200 0.0496133 0.0456861 2026-02-17T04:56:53.003 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:53.001+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x6616d00 msgr2=0x7f6e84085330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:53.937 INFO:teuthology.orchestra.run.trial100.stdout: 4 16 1292 1276 1275.91 932 0.0429029 0.0499113 2026-02-17T04:56:54.090 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:54.089+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:54.090 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:54.089+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x6616d00 msgr2=0x7f6e840846e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:54.371 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:54.370+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x6616d00 msgr2=0x7f6e840846e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:54.812 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:54.811+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x6616d00 msgr2=0x7f6e84081aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:54.936 INFO:teuthology.orchestra.run.trial100.stdout: 5 16 1592 1576 1260.71 1200 0.0413361 0.0505612 2026-02-17T04:56:55.547 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:55.546+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:55.791 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:55.789+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:55.937 INFO:teuthology.orchestra.run.trial100.stdout: 6 16 1971 1955 1303.24 1516 0.0428904 0.0489447 2026-02-17T04:56:56.406 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:56.404+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e840848d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:56.491 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:56.489+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e840848d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:56.567 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:56.566+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:56.740 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:56.739+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:56.936 INFO:teuthology.orchestra.run.trial100.stdout: 7 16 2341 2325 1328.48 1480 0.0357647 0.0480316 2026-02-17T04:56:57.282 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:57.281+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e84084870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:57.584 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:57.583+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:57.937 INFO:teuthology.orchestra.run.trial100.stdout: 8 16 2711 2695 1347.41 1480 0.0379207 0.0473831 2026-02-17T04:56:58.262 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:58.261+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e84084870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:58.281 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:58.280+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e84084870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:58.505 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:58.504+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e84084870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:58.530 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:58.530+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:58.794 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:58.792+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e84084870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:58.937 INFO:teuthology.orchestra.run.trial100.stdout: 9 16 3008 2992 1329.69 1188 0.0432118 0.0479838 2026-02-17T04:56:59.316 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:59.315+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:59.380 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:59.380+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:59.419 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:59.418+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:56:59.690 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:59.689+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e84084870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:56:59.936 INFO:teuthology.orchestra.run.trial100.stdout: 10 16 3307 3291 1316.31 1196 0.036757 0.0485231 2026-02-17T04:56:59.984 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:56:59.983+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e84084870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:00.053 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:00.052+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:00.063 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:00.062+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e84084870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:00.470 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:00.469+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e84084870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:00.575 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:00.574+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840610c0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:00.937 INFO:teuthology.orchestra.run.trial100.stdout: 11 16 3688 3672 1335.18 1524 0.0429581 0.0478408 2026-02-17T04:57:01.787 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:01.786+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081aa0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:01.937 INFO:teuthology.orchestra.run.trial100.stdout: 12 16 3952 3936 1311.91 1056 0.0427628 0.048184 2026-02-17T04:57:02.297 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:02.296+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e840844a0 msgr2=0x7f6e84084870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:02.323 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:02.322+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081aa0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:02.937 INFO:teuthology.orchestra.run.trial100.stdout: 13 16 4307 4291 1320.22 1420 0.0379365 0.0484029 2026-02-17T04:57:03.259 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:03.257+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081aa0 msgr2=0x7f6e840814a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:03.558 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:03.557+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:03.937 INFO:teuthology.orchestra.run.trial100.stdout: 14 16 4521 4505 1287.05 856 0.0410476 0.0496582 2026-02-17T04:57:04.206 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:04.204+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:04.568 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:04.566+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:04.611 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:04.609+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:04.937 INFO:teuthology.orchestra.run.trial100.stdout: 15 16 4901 4885 1302.58 1520 0.0400707 0.049062 2026-02-17T04:57:05.937 INFO:teuthology.orchestra.run.trial100.stdout: 16 16 5222 5206 1301.41 1284 0.0441235 0.0491047 2026-02-17T04:57:06.937 INFO:teuthology.orchestra.run.trial100.stdout: 17 16 5378 5362 1261.56 624 0.633342 0.04978 2026-02-17T04:57:07.928 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:07.926+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:07.937 INFO:teuthology.orchestra.run.trial100.stdout: 18 16 5773 5757 1279.24 1580 0.0415824 0.0499611 2026-02-17T04:57:08.937 INFO:teuthology.orchestra.run.trial100.stdout: 19 16 6164 6148 1294.22 1564 0.0380482 0.0493988 2026-02-17T04:57:09.358 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:09.356+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:09.425 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:09.423+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:09.515 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:09.513+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:09.906 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:09.904+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:09.937 INFO:teuthology.orchestra.run.trial100.stdout:2026-02-17T04:57:09.937064+0000 min lat: 0.010245 max lat: 0.651254 avg lat: 0.0490063 2026-02-17T04:57:09.938 INFO:teuthology.orchestra.run.trial100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T04:57:09.938 INFO:teuthology.orchestra.run.trial100.stdout: 20 16 6539 6523 1304.51 1500 0.0451788 0.0490063 2026-02-17T04:57:10.325 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:10.323+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:10.372 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:10.370+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:10.754 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:10.752+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:10.768 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:10.766+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:10.816 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:10.814+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:10.937 INFO:teuthology.orchestra.run.trial100.stdout: 21 16 6907 6891 1312.48 1472 0.0399752 0.0487154 2026-02-17T04:57:11.938 INFO:teuthology.orchestra.run.trial100.stdout: 22 16 7291 7275 1322.63 1536 0.0429643 0.0483239 2026-02-17T04:57:12.121 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:12.119+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:12.219 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:12.217+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:12.938 INFO:teuthology.orchestra.run.trial100.stdout: 23 16 7670 7654 1331.04 1516 0.039276 0.0480368 2026-02-17T04:57:13.195 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:13.192+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:13.609 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:13.607+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:13.938 INFO:teuthology.orchestra.run.trial100.stdout: 24 16 7983 7967 1327.74 1252 0.0358974 0.0481647 2026-02-17T04:57:14.154 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:14.152+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:14.649 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:14.646+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:14.868 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:14.865+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:14.938 INFO:teuthology.orchestra.run.trial100.stdout: 25 16 8367 8351 1336.07 1536 0.0433933 0.0478609 2026-02-17T04:57:14.945 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:14.943+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:15.832 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:15.830+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:15.938 INFO:teuthology.orchestra.run.trial100.stdout: 26 16 8739 8723 1341.9 1488 0.0475813 0.0476504 2026-02-17T04:57:16.175 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:16.173+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:16.199 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:16.197+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:16.538 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:16.536+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:16.938 INFO:teuthology.orchestra.run.trial100.stdout: 27 16 8974 8958 1327.02 940 0.0415404 0.048189 2026-02-17T04:57:17.010 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:17.008+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:17.938 INFO:teuthology.orchestra.run.trial100.stdout: 28 16 9191 9175 1310.62 868 0.0410416 0.0480645 2026-02-17T04:57:18.274 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:18.272+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:18.938 INFO:teuthology.orchestra.run.trial100.stdout: 29 16 9498 9482 1307.77 1228 0.0447518 0.0489001 2026-02-17T04:57:19.264 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:19.261+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:19.323 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:19.320+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:19.497 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:19.495+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:19.938 INFO:teuthology.orchestra.run.trial100.stdout: 30 16 9779 9763 1301.64 1124 0.0402633 0.0491324 2026-02-17T04:57:20.623 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:20.621+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:20.665 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:20.663+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84082230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:20.938 INFO:teuthology.orchestra.run.trial100.stdout: 31 16 10162 10146 1309.07 1532 0.0416872 0.0488522 2026-02-17T04:57:21.319 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:21.317+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84081e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:21.938 INFO:teuthology.orchestra.run.trial100.stdout: 32 16 10412 10396 1299.41 1000 0.0488052 0.0490089 2026-02-17T04:57:22.938 INFO:teuthology.orchestra.run.trial100.stdout: 33 16 10775 10759 1304.03 1452 0.0441499 0.0490372 2026-02-17T04:57:23.880 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:23.877+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:23.909 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:23.907+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84061490 msgr2=0x7f6e84081e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:23.938 INFO:teuthology.orchestra.run.trial100.stdout: 34 16 11146 11130 1309.32 1484 0.0531387 0.0488453 2026-02-17T04:57:24.539 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:24.537+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:24.539 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:24.537+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:24.675 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:24.672+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:24.857 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:24.855+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:24.939 INFO:teuthology.orchestra.run.trial100.stdout: 35 16 11521 11505 1314.76 1500 0.0460482 0.0486472 2026-02-17T04:57:25.041 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:25.039+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:25.123 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:25.121+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:25.207 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:25.205+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:25.939 INFO:teuthology.orchestra.run.trial100.stdout: 36 16 11755 11739 1304.24 936 0.0379851 0.0490415 2026-02-17T04:57:26.626 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:26.624+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:26.938 INFO:teuthology.orchestra.run.trial100.stdout: 37 16 12056 12040 1301.53 1204 0.044065 0.0491393 2026-02-17T04:57:27.939 INFO:teuthology.orchestra.run.trial100.stdout: 38 16 12364 12348 1299.7 1232 0.0362598 0.0492159 2026-02-17T04:57:28.939 INFO:teuthology.orchestra.run.trial100.stdout: 39 16 12667 12651 1297.45 1212 0.0379973 0.0492977 2026-02-17T04:57:29.750 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:29.748+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:29.939 INFO:teuthology.orchestra.run.trial100.stdout:2026-02-17T04:57:29.938451+0000 min lat: 0.0063652 max lat: 0.653685 avg lat: 0.0491014 2026-02-17T04:57:29.939 INFO:teuthology.orchestra.run.trial100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T04:57:29.939 INFO:teuthology.orchestra.run.trial100.stdout: 40 16 13043 13027 1302.61 1504 0.0437533 0.0491014 2026-02-17T04:57:30.066 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:30.064+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:30.889 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:30.887+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:30.939 INFO:teuthology.orchestra.run.trial100.stdout: 41 16 13342 13326 1300.01 1196 0.0398567 0.0492084 2026-02-17T04:57:31.139 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:31.137+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:31.343 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:31.340+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:31.445 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:31.443+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:31.907 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:31.904+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:31.939 INFO:teuthology.orchestra.run.trial100.stdout: 42 16 13701 13685 1303.24 1436 0.0443597 0.04908 2026-02-17T04:57:32.034 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:32.031+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:32.323 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:32.321+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e840a7200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:32.375 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:32.373+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:32.401 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:32.398+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e840a7200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:32.802 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:32.800+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e840a6150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:32.856 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:32.854+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e840a5c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:32.939 INFO:teuthology.orchestra.run.trial100.stdout: 43 16 14070 14054 1307.26 1476 0.0399524 0.04893 2026-02-17T04:57:33.090 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:33.088+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e840825e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:33.755 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:33.752+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:33.812 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:33.810+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e840825e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:33.939 INFO:teuthology.orchestra.run.trial100.stdout: 44 16 14454 14438 1312.45 1536 0.0377148 0.048739 2026-02-17T04:57:34.147 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:34.145+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:34.939 INFO:teuthology.orchestra.run.trial100.stdout: 45 16 14818 14802 1315.64 1456 0.0382449 0.0486217 2026-02-17T04:57:35.205 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:35.203+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:35.939 INFO:teuthology.orchestra.run.trial100.stdout: 46 16 15193 15177 1319.65 1500 0.0446909 0.0484696 2026-02-17T04:57:36.059 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:36.057+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084870 msgr2=0x7f6e84084c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:36.680 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:36.678+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:36.939 INFO:teuthology.orchestra.run.trial100.stdout: 47 16 15504 15488 1318.03 1244 0.0458613 0.0485302 2026-02-17T04:57:37.389 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:37.387+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:37.437 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:37.434+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:37.714 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:37.712+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:37.939 INFO:teuthology.orchestra.run.trial100.stdout: 48 15 15734 15719 1309.82 924 0.24152 0.0487491 2026-02-17T04:57:38.090 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:38.088+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:38.427 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:38.425+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:38.525 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:38.523+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:38.940 INFO:teuthology.orchestra.run.trial100.stdout: 49 16 16027 16011 1306.93 1168 0.0431965 0.0489411 2026-02-17T04:57:39.010 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:39.008+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:39.277 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:39.275+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:39.456 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:39.455+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:39.650 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:39.648+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:39.650 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:39.649+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:39.939 INFO:teuthology.orchestra.run.trial100.stdout: 50 16 16329 16313 1304.95 1208 0.0379522 0.0490249 2026-02-17T04:57:40.100 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:40.099+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:40.160 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:40.159+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:40.940 INFO:teuthology.orchestra.run.trial100.stdout: 51 16 16694 16678 1307.99 1460 0.0405682 0.0489091 2026-02-17T04:57:41.353 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:41.351+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:41.353 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:41.352+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:41.749 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:41.742+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:41.894 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:41.893+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:41.940 INFO:teuthology.orchestra.run.trial100.stdout: 52 16 16911 16895 1299.52 868 0.0399363 0.0491868 2026-02-17T04:57:42.099 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:42.098+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:42.922 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:42.921+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:42.940 INFO:teuthology.orchestra.run.trial100.stdout: 53 16 17225 17209 1298.7 1256 0.0492866 0.0492489 2026-02-17T04:57:43.092 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:43.091+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:43.211 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:43.209+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:43.490 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:43.489+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:43.546 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:43.545+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:43.627 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:43.626+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:43.641 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:43.640+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:43.651 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:43.650+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:43.904 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:43.903+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:43.940 INFO:teuthology.orchestra.run.trial100.stdout: 54 16 17494 17478 1294.57 1076 0.0541444 0.0494059 2026-02-17T04:57:44.502 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:44.501+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:44.788 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:44.786+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:44.940 INFO:teuthology.orchestra.run.trial100.stdout: 55 16 17789 17773 1292.49 1180 0.0480447 0.0494923 2026-02-17T04:57:45.940 INFO:teuthology.orchestra.run.trial100.stdout: 56 16 18168 18152 1296.48 1516 0.0400089 0.049345 2026-02-17T04:57:46.050 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:46.049+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:46.203 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:46.202+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:46.436 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:46.435+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:46.940 INFO:teuthology.orchestra.run.trial100.stdout: 57 16 18474 18458 1295.21 1224 0.0369335 0.0493952 2026-02-17T04:57:46.991 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:46.990+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:46.991 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:46.990+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:47.451 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:47.450+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:47.517 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:47.516+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:47.814 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:47.813+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:47.940 INFO:teuthology.orchestra.run.trial100.stdout: 58 16 18673 18657 1286.6 796 0.0442887 0.0496543 2026-02-17T04:57:48.202 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:48.201+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84081e70 msgr2=0x7f6e84082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-02-17T04:57:48.940 INFO:teuthology.orchestra.run.trial100.stdout: 59 16 19001 18985 1287.03 1312 0.0529591 0.0497021 2026-02-17T04:57:49.940 INFO:teuthology.orchestra.run.trial100.stdout:2026-02-17T04:57:49.939888+0000 min lat: 0.0063652 max lat: 0.653685 avg lat: 0.0497205 2026-02-17T04:57:49.941 INFO:teuthology.orchestra.run.trial100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T04:57:49.941 INFO:teuthology.orchestra.run.trial100.stdout: 60 12 19314 19302 1286.71 1268 0.0394298 0.0497205 2026-02-17T04:57:50.941 INFO:teuthology.orchestra.run.trial100.stdout:Total time run: 60.0278 2026-02-17T04:57:50.941 INFO:teuthology.orchestra.run.trial100.stdout:Total writes made: 19314 2026-02-17T04:57:50.941 INFO:teuthology.orchestra.run.trial100.stdout:Write size: 4194304 2026-02-17T04:57:50.941 INFO:teuthology.orchestra.run.trial100.stdout:Object size: 4194304 2026-02-17T04:57:50.941 INFO:teuthology.orchestra.run.trial100.stdout:Bandwidth (MB/sec): 1287 2026-02-17T04:57:50.941 INFO:teuthology.orchestra.run.trial100.stdout:Stddev Bandwidth: 237.05 2026-02-17T04:57:50.941 INFO:teuthology.orchestra.run.trial100.stdout:Max bandwidth (MB/sec): 1580 2026-02-17T04:57:50.941 INFO:teuthology.orchestra.run.trial100.stdout:Min bandwidth (MB/sec): 624 2026-02-17T04:57:50.941 INFO:teuthology.orchestra.run.trial100.stdout:Average IOPS: 321 2026-02-17T04:57:50.942 INFO:teuthology.orchestra.run.trial100.stdout:Stddev IOPS: 59.2625 2026-02-17T04:57:50.942 INFO:teuthology.orchestra.run.trial100.stdout:Max IOPS: 395 2026-02-17T04:57:50.942 INFO:teuthology.orchestra.run.trial100.stdout:Min IOPS: 156 2026-02-17T04:57:50.942 INFO:teuthology.orchestra.run.trial100.stdout:Average Latency(s): 0.0497138 2026-02-17T04:57:50.942 INFO:teuthology.orchestra.run.trial100.stdout:Stddev Latency(s): 0.0436773 2026-02-17T04:57:50.942 INFO:teuthology.orchestra.run.trial100.stdout:Max latency(s): 0.653685 2026-02-17T04:57:50.942 INFO:teuthology.orchestra.run.trial100.stdout:Min latency(s): 0.0063652 2026-02-17T04:57:50.943 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:57:50.940+0000 7f6ea6a13640 0 -- 10.20.193.100:0/1524025338 >> v2:10.20.193.100:6804/3637649603 conn(0x7f6e84084c40 msgr2=0x7f6e84082780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-02-17T04:57:50.947 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd df name osd.3 --format=json 2026-02-17T04:57:51.166 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:57:51.166 INFO:teuthology.orchestra.run.trial100.stdout:{"nodes":[{"id":3,"device_class":"ssd","name":"osd.3","type":"osd","type_id":0,"crush_weight":0.087890625,"depth":2,"pool_weights":{},"reweight":1,"kb":94371840,"kb_used":38764368,"kb_used_data":38142144,"kb_used_omap":13,"kb_used_meta":622194,"kb_avail":55607472,"utilization":41.076202392578125,"var":1,"pgs":1,"status":"up"}],"stray":[],"summary":{"total_kb":94371840,"total_kb_used":38764368,"total_kb_used_data":38142144,"total_kb_used_omap":13,"total_kb_used_meta":622194,"total_kb_avail":55607472,"average_utilization":41.076202392578125,"min_var":1,"max_var":1,"dev":0}} 2026-02-17T04:57:51.175 DEBUG:tasks.backfill_toofull:primary osd df: {'id': 3, 'device_class': 'ssd', 'name': 'osd.3', 'type': 'osd', 'type_id': 0, 'crush_weight': 0.087890625, 'depth': 2, 'pool_weights': {}, 'reweight': 1, 'kb': 94371840, 'kb_used': 38764368, 'kb_used_data': 38142144, 'kb_used_omap': 13, 'kb_used_meta': 622194, 'kb_avail': 55607472, 'utilization': 41.076202392578125, 'var': 1, 'pgs': 1, 'status': 'up'} 2026-02-17T04:57:51.176 INFO:tasks.backfill_toofull:test backfill reservation rejected with toofull 2026-02-17T04:57:51.176 DEBUG:tasks.backfill_toofull:update nearfull ratio to 0.33271723937988285 and backfillfull ratio to 0.3696858215332032 2026-02-17T04:57:51.176 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-nearfull-ratio 0.334 2026-02-17T04:57:51.383 INFO:teuthology.orchestra.run.trial100.stderr:osd set-nearfull-ratio 0.334 2026-02-17T04:57:51.395 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-backfillfull-ratio 0.371 2026-02-17T04:57:52.388 INFO:teuthology.orchestra.run.trial100.stderr:osd set-backfillfull-ratio 0.371 2026-02-17T04:57:52.401 DEBUG:tasks.backfill_toofull:start tartget osd 1 2026-02-17T04:57:52.401 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-17T04:57:52.402 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-17T04:57:52.404 INFO:tasks.ceph.osd.1:Started 2026-02-17T04:57:52.404 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:no valid command found; 10 closest matches: 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:0 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:1 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:2 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:abort 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:assert 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:bluestore collections 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:bluestore compression stats [] 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:bluestore list [] [] 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:bluestore onode metadata 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:bluestore show sharding 2026-02-17T04:57:52.469 INFO:teuthology.orchestra.run.trial100.stderr:admin_socket: invalid command 2026-02-17T04:57:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T04:57:52.471 INFO:tasks.backfill_toofull.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-17T04:57:52.600 INFO:tasks.ceph.osd.1.trial100.stderr:2026-02-17T04:57:52.598+0000 7fa88f611780 -1 Falling back to public interface 2026-02-17T04:57:57.471 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-17T04:57:57.541 INFO:teuthology.orchestra.run.trial100.stderr:no valid command found; 10 closest matches: 2026-02-17T04:57:57.541 INFO:teuthology.orchestra.run.trial100.stderr:0 2026-02-17T04:57:57.541 INFO:teuthology.orchestra.run.trial100.stderr:1 2026-02-17T04:57:57.541 INFO:teuthology.orchestra.run.trial100.stderr:2 2026-02-17T04:57:57.541 INFO:teuthology.orchestra.run.trial100.stderr:abort 2026-02-17T04:57:57.541 INFO:teuthology.orchestra.run.trial100.stderr:assert 2026-02-17T04:57:57.541 INFO:teuthology.orchestra.run.trial100.stderr:bluefs debug_inject_read_zeros 2026-02-17T04:57:57.541 INFO:teuthology.orchestra.run.trial100.stderr:bluefs files list 2026-02-17T04:57:57.541 INFO:teuthology.orchestra.run.trial100.stderr:bluefs stats 2026-02-17T04:57:57.541 INFO:teuthology.orchestra.run.trial100.stderr:bluestore allocator dump block 2026-02-17T04:57:57.542 INFO:teuthology.orchestra.run.trial100.stderr:bluestore allocator fragmentation block 2026-02-17T04:57:57.542 INFO:teuthology.orchestra.run.trial100.stderr:admin_socket: invalid command 2026-02-17T04:57:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T04:57:57.543 INFO:tasks.backfill_toofull.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-17T04:57:58.243 INFO:tasks.ceph.osd.1.trial100.stderr:2026-02-17T04:57:58.241+0000 7fa88f611780 -1 osd.1 21 log_to_monitors true 2026-02-17T04:57:58.446 INFO:tasks.ceph.osd.1.trial100.stderr:2026-02-17T04:57:58.444+0000 7fa88292a640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T04:58:02.544 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-17T04:58:02.612 INFO:teuthology.orchestra.run.trial100.stdout:{ 2026-02-17T04:58:02.612 INFO:teuthology.orchestra.run.trial100.stdout: "ops": [], 2026-02-17T04:58:02.613 INFO:teuthology.orchestra.run.trial100.stdout: "num_ops": 0 2026-02-17T04:58:02.613 INFO:teuthology.orchestra.run.trial100.stdout:} 2026-02-17T04:58:02.617 INFO:tasks.backfill_toofull.ceph_manager:waiting for peering to complete 2026-02-17T04:58:02.618 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:02.793 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:02.793 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:02.801 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2026-02-17T04:58:02.289946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511564,"kb_used_data":141196748,"kb_used_omap":46,"kb_used_meta":1314705,"kb_avail":234975796,"statfs":{"total":386547056640,"available":240615215104,"internally_reserved":0,"allocated":144585469952,"data_stored":144584904080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47681,"internal_metadata":1346258367},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000551"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79972,"reported_epoch":26,"state":"active+undersized+degraded","last_fresh":"2026-02-17T04:57:54.389731+0000","last_change":"2026-02-17T04:56:49.950002+0000","last_active":"2026-02-17T04:57:54.389731+0000","last_peered":"2026-02-17T04:57:54.389731+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:56:21.431181+0000","last_became_peered":"2026-02-17T04:56:21.431181+0000","last_unstale":"2026-02-17T04:57:54.389731+0000","last_undegraded":"2026-02-17T04:56:49.949869+0000","last_fullsized":"2026-02-17T04:56:21.426703+0000","mapping_epoch":22,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:16:29.071893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":47,"reported_epoch":26,"state":"active+undersized+degraded","last_fresh":"2026-02-17T04:57:54.389647+0000","last_change":"2026-02-17T04:56:21.426802+0000","last_active":"2026-02-17T04:57:54.389647+0000","last_peered":"2026-02-17T04:57:54.389647+0000","last_clean":"2026-02-17T04:54:00.476997+0000","last_became_active":"2026-02-17T04:56:21.426802+0000","last_became_peered":"2026-02-17T04:56:21.426802+0000","last_unstale":"2026-02-17T04:57:54.389647+0000","last_undegraded":"2026-02-17T04:56:21.426315+0000","last_fullsized":"2026-02-17T04:56:21.426255+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T15:54:21.464716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177516,"kb_used_data":39555292,"kb_used_omap":14,"kb_used_meta":622193,"kb_avail":54194324,"statfs":{"total":96636764160,"available":55494987776,"internally_reserved":0,"allocated":40504619008,"data_stored":40504473713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15294,"internal_metadata":637125698},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177516,"kb_used_data":39555292,"kb_used_omap":13,"kb_used_meta":622194,"kb_avail":54194324,"statfs":{"total":96636764160,"available":55494987776,"internally_reserved":0,"allocated":40504619008,"data_stored":40504473713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13994,"internal_metadata":637126998},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129032,"kb_used_data":62085624,"kb_used_omap":4,"kb_used_meta":43387,"kb_avail":32242808,"statfs":{"total":96636764160,"available":33016635392,"internally_reserved":0,"allocated":63575678976,"data_stored":63575552573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":44429009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":540,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":552960,"data_stored":404081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13994,"internal_metadata":27576662},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:02.802 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:02.976 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:02.977 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:02.985 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2026-02-17T04:58:02.289946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511564,"kb_used_data":141196748,"kb_used_omap":46,"kb_used_meta":1314705,"kb_avail":234975796,"statfs":{"total":386547056640,"available":240615215104,"internally_reserved":0,"allocated":144585469952,"data_stored":144584904080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47681,"internal_metadata":1346258367},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000551"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79972,"reported_epoch":26,"state":"active+undersized+degraded","last_fresh":"2026-02-17T04:57:54.389731+0000","last_change":"2026-02-17T04:56:49.950002+0000","last_active":"2026-02-17T04:57:54.389731+0000","last_peered":"2026-02-17T04:57:54.389731+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:56:21.431181+0000","last_became_peered":"2026-02-17T04:56:21.431181+0000","last_unstale":"2026-02-17T04:57:54.389731+0000","last_undegraded":"2026-02-17T04:56:49.949869+0000","last_fullsized":"2026-02-17T04:56:21.426703+0000","mapping_epoch":22,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:16:29.071893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":47,"reported_epoch":26,"state":"active+undersized+degraded","last_fresh":"2026-02-17T04:57:54.389647+0000","last_change":"2026-02-17T04:56:21.426802+0000","last_active":"2026-02-17T04:57:54.389647+0000","last_peered":"2026-02-17T04:57:54.389647+0000","last_clean":"2026-02-17T04:54:00.476997+0000","last_became_active":"2026-02-17T04:56:21.426802+0000","last_became_peered":"2026-02-17T04:56:21.426802+0000","last_unstale":"2026-02-17T04:57:54.389647+0000","last_undegraded":"2026-02-17T04:56:21.426315+0000","last_fullsized":"2026-02-17T04:56:21.426255+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T15:54:21.464716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177516,"kb_used_data":39555292,"kb_used_omap":14,"kb_used_meta":622193,"kb_avail":54194324,"statfs":{"total":96636764160,"available":55494987776,"internally_reserved":0,"allocated":40504619008,"data_stored":40504473713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15294,"internal_metadata":637125698},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177516,"kb_used_data":39555292,"kb_used_omap":13,"kb_used_meta":622194,"kb_avail":54194324,"statfs":{"total":96636764160,"available":55494987776,"internally_reserved":0,"allocated":40504619008,"data_stored":40504473713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13994,"internal_metadata":637126998},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129032,"kb_used_data":62085624,"kb_used_omap":4,"kb_used_meta":43387,"kb_avail":32242808,"statfs":{"total":96636764160,"available":33016635392,"internally_reserved":0,"allocated":63575678976,"data_stored":63575552573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":44429009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":540,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":552960,"data_stored":404081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13994,"internal_metadata":27576662},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:02.986 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:03.201 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:03.201 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 backfillfull osd(s)","count":2},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19317/57949 objects degraded (33.334%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":250,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":30,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":2}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145931841536,"bytes_avail":240615215104,"bytes_total":386547056640,"degraded_objects":19317,"degraded_total":57949,"degraded_ratio":0.33334483770211742},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":2,"modified":"2026-02-17T04:54:58.264859+0000","services":{}},"progress_events":{}} 2026-02-17T04:58:03.209 INFO:tasks.backfill_toofull.ceph_manager:{'fsid': 'ac0ddb8a-2bd5-421d-9307-ad42ed106d65', 'health': {'status': 'HEALTH_WARN', 'checks': {'OSDMAP_FLAGS': {'severity': 'HEALTH_WARN', 'summary': {'message': 'noout,nobackfill,norecover flag(s) set', 'count': 26}, 'muted': False}, 'OSD_BACKFILLFULL': {'severity': 'HEALTH_WARN', 'summary': {'message': '2 backfillfull osd(s)', 'count': 2}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 19317/57949 objects degraded (33.334%), 2 pgs degraded, 2 pgs undersized', 'count': 4}, 'muted': False}, 'POOL_BACKFILLFULL': {'severity': 'HEALTH_WARN', 'summary': {'message': '2 pool(s) backfillfull', 'count': 2}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 250, 'monmap': {'epoch': 1, 'min_mon_release_name': 'tentacle', 'num_mons': 3}, 'osdmap': {'epoch': 30, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1771304279, 'num_in_osds': 4, 'osd_in_since': 1771304034, 'num_remapped_pgs': 1}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+undersized+degraded', 'count': 2}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 19317, 'data_bytes': 81009115672, 'bytes_used': 145931841536, 'bytes_avail': 240615215104, 'bytes_total': 386547056640, 'degraded_objects': 19317, 'degraded_total': 57949, 'degraded_ratio': 0.3333448377021174}, 'fsmap': {'epoch': 1, 'btime': '2026-02-17T04:53:53:064030+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs'], 'services': {}}, 'servicemap': {'epoch': 2, 'modified': '2026-02-17T04:54:58.264859+0000', 'services': {}}, 'progress_events': {}} 2026-02-17T04:58:03.209 INFO:tasks.backfill_toofull.ceph_manager:active! 2026-02-17T04:58:03.209 INFO:tasks.backfill_toofull.ceph_manager:waiting for osd.1 to be up 2026-02-17T04:58:03.209 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T04:58:03.475 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:58:03.473+0000 7f0539d53640 0 --2- 10.20.193.100:0/1907162464 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7f0534178ca0 0x7f0534199080 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=8382015672080204288 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T04:58:03.790 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:03.790 INFO:teuthology.orchestra.run.trial100.stdout:{"epoch":30,"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","created":"2026-02-17T04:53:53.067772+0000","modified":"2026-02-17T04:58:01.448903+0000","last_up_change":"2026-02-17T04:57:59.440995+0000","last_in_change":"2026-02-17T04:53:54.817146+0000","flags":"noout,nobackfill,norecover,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801600,"flags_set":["nobackfill","noout","norecover","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.37099999189376831,"nearfull_ratio":0.33399999141693115,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T04:53:58.268259+0000","flags":4097,"flags_names":"hashpspool,backfillfull","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T04:54:07.279692+0000","flags":4097,"flags_names":"hashpspool,backfillfull","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_toofull","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"f589f588-f611-4e63-a4a6-8124a0132ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6805","nonce":119222286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6808","nonce":119222286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6813","nonce":119222286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6811","nonce":119222286}]},"public_addr":"10.20.193.100:6805/119222286","cluster_addr":"10.20.193.100:6808/119222286","heartbeat_back_addr":"10.20.193.100:6813/119222286","heartbeat_front_addr":"10.20.193.100:6811/119222286","state":["exists","up"]},{"osd":1,"uuid":"6d699c04-f7f0-4e53-bbbf-a2cece5b5525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":21,"up_from":28,"up_thru":28,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6800","nonce":1474281622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6801","nonce":1474281622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6803","nonce":1474281622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6802","nonce":1474281622}]},"public_addr":"10.20.193.100:6800/1474281622","cluster_addr":"10.20.193.100:6801/1474281622","heartbeat_back_addr":"10.20.193.100:6803/1474281622","heartbeat_front_addr":"10.20.193.100:6802/1474281622","state":["exists","up"]},{"osd":2,"uuid":"ff2abd17-d4b9-4c16-8108-10b9a52f0c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6806","nonce":3399893774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6809","nonce":3399893774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6815","nonce":3399893774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6812","nonce":3399893774}]},"public_addr":"10.20.193.100:6806/3399893774","cluster_addr":"10.20.193.100:6809/3399893774","heartbeat_back_addr":"10.20.193.100:6815/3399893774","heartbeat_front_addr":"10.20.193.100:6812/3399893774","state":["backfillfull","exists","up"]},{"osd":3,"uuid":"6334acee-00c0-4ca3-8935-a1dfefa95e1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6804","nonce":3637649603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6807","nonce":3637649603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6814","nonce":3637649603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.100:6810","nonce":3637649603}]},"public_addr":"10.20.193.100:6804/3637649603","cluster_addr":"10.20.193.100:6807/3637649603","heartbeat_back_addr":"10.20.193.100:6814/3637649603","heartbeat_front_addr":"10.20.193.100:6810/3637649603","state":["backfillfull","exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T04:53:56.795492+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T04:53:56.802962+0000","dead_epoch":22},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T04:53:56.762718+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T04:53:56.823375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"2.0","osds":[3,2147483647,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_toofull":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T04:58:03.800 INFO:tasks.backfill_toofull.ceph_manager:osd.1 is up 2026-02-17T04:58:03.800 DEBUG:tasks.backfill_toofull:waiting for pg 2.0 state is backfill_toofull 2026-02-17T04:58:08.802 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T04:58:08.802 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T04:58:08.803 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T04:58:08.803 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T04:58:08.889 INFO:teuthology.orchestra.run.trial100.stdout:120259084292 2026-02-17T04:58:08.890 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T04:58:08.894 INFO:teuthology.orchestra.run.trial100.stdout:38654705719 2026-02-17T04:58:08.894 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T04:58:08.895 INFO:teuthology.orchestra.run.trial100.stdout:38654705719 2026-02-17T04:58:08.896 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T04:58:08.898 INFO:teuthology.orchestra.run.trial100.stdout:38654705719 2026-02-17T04:58:08.898 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T04:58:09.116 INFO:teuthology.orchestra.run.trial100.stdout:38654705718 2026-02-17T04:58:09.116 INFO:teuthology.orchestra.run.trial100.stdout:120259084290 2026-02-17T04:58:09.119 INFO:teuthology.orchestra.run.trial100.stdout:38654705718 2026-02-17T04:58:09.128 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705719 got 38654705718 for osd.2 2026-02-17T04:58:09.129 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084292 got 120259084290 for osd.1 2026-02-17T04:58:09.131 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705719 got 38654705718 for osd.0 2026-02-17T04:58:09.157 INFO:teuthology.orchestra.run.trial100.stdout:38654705717 2026-02-17T04:58:09.167 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705719 got 38654705717 for osd.3 2026-02-17T04:58:10.129 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T04:58:10.130 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T04:58:10.131 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T04:58:10.168 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T04:58:10.308 INFO:teuthology.orchestra.run.trial100.stdout:38654705718 2026-02-17T04:58:10.310 INFO:teuthology.orchestra.run.trial100.stdout:38654705718 2026-02-17T04:58:10.311 INFO:teuthology.orchestra.run.trial100.stdout:120259084290 2026-02-17T04:58:10.320 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705719 got 38654705718 for osd.2 2026-02-17T04:58:10.321 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705719 got 38654705718 for osd.0 2026-02-17T04:58:10.322 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084292 got 120259084290 for osd.1 2026-02-17T04:58:10.352 INFO:teuthology.orchestra.run.trial100.stdout:38654705717 2026-02-17T04:58:10.362 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705719 got 38654705717 for osd.3 2026-02-17T04:58:11.321 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T04:58:11.322 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T04:58:11.322 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T04:58:11.363 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T04:58:11.499 INFO:teuthology.orchestra.run.trial100.stdout:38654705719 2026-02-17T04:58:11.500 INFO:teuthology.orchestra.run.trial100.stdout:120259084292 2026-02-17T04:58:11.502 INFO:teuthology.orchestra.run.trial100.stdout:38654705719 2026-02-17T04:58:11.511 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084292 got 120259084292 for osd.1 2026-02-17T04:58:11.511 DEBUG:teuthology.parallel:result is None 2026-02-17T04:58:11.512 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705719 got 38654705719 for osd.2 2026-02-17T04:58:11.512 DEBUG:teuthology.parallel:result is None 2026-02-17T04:58:11.514 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705719 got 38654705719 for osd.0 2026-02-17T04:58:11.514 DEBUG:teuthology.parallel:result is None 2026-02-17T04:58:11.631 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:58:11.630+0000 7f681ffff640 0 --2- 10.20.193.100:0/293531743 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7f682805c490 0x7f6828137150 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T04:58:11.742 INFO:teuthology.orchestra.run.trial100.stdout:38654705719 2026-02-17T04:58:11.752 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705719 got 38654705719 for osd.3 2026-02-17T04:58:11.752 DEBUG:teuthology.parallel:result is None 2026-02-17T04:58:11.752 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:11.927 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:11.927 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:11.936 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2026-02-17T04:58:10.290975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.840822"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79988,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460653+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460653+0000","last_peered":"2026-02-17T04:58:04.460653+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460653+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:11.936 INFO:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79988, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:04.460653+0000', 'last_change': '2026-02-17T04:58:01.810816+0000', 'last_active': '2026-02-17T04:58:04.460653+0000', 'last_peered': '2026-02-17T04:58:04.460653+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:04.460653+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:11.936 INFO:tasks.backfill_toofull:test pg not enter backfill_toofull after restarting backfill 2026-02-17T04:58:11.936 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df detail --format=json 2026-02-17T04:58:12.151 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:12.151 INFO:teuthology.orchestra.run.trial100.stdout:{"stats":{"total_bytes":386547056640,"total_avail_bytes":240615010304,"total_used_bytes":145932046336,"total_used_raw_bytes":145932046336,"total_used_raw_ratio":0.37752723693847656,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4},"stats_by_class":{"ssd":{"total_bytes":386547056640,"total_avail_bytes":240615010304,"total_used_bytes":145932046336,"total_used_raw_bytes":145932046336,"total_used_raw_ratio":0.37752723693847656}},"pools":[{"name":".mgr","id":1,"stats":{"stored":328192,"stored_data":328192,"stored_omap":0,"objects":2,"kb_used":648,"bytes_used":663552,"data_bytes_used":663552,"omap_bytes_used":0,"percent_used":5.8857090152741875e-06,"max_avail":56369426432,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":26,"rd_bytes":19456,"wr":37,"wr_bytes":329728,"compress_bytes_used":0,"compress_under_bytes":0,"stored_raw":656384,"avail_raw":112738856448}},{"name":"unique_pool_0","id":2,"stats":{"stored":96389308416,"stored_data":96389308416,"stored_omap":0,"objects":19315,"kb_used":141195276,"bytes_used":144583962624,"data_bytes_used":144583962624,"omap_bytes_used":0,"percent_used":0.5618777871131897,"max_avail":75159232512,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":1,"rd_bytes":1024,"wr":79947,"wr_bytes":208159115264,"compress_bytes_used":0,"compress_under_bytes":0,"stored_raw":144583966720,"avail_raw":112738856448}}]} 2026-02-17T04:58:12.160 DEBUG:tasks.backfill_toofull:pool unique_pool_0 df: {'stored': 96389308416, 'stored_data': 96389308416, 'stored_omap': 0, 'objects': 19315, 'kb_used': 141195276, 'bytes_used': 144583962624, 'data_bytes_used': 144583962624, 'omap_bytes_used': 0, 'percent_used': 0.5618777871131897, 'max_avail': 75159232512, 'quota_objects': 0, 'quota_bytes': 0, 'dirty': 0, 'rd': 1, 'rd_bytes': 1024, 'wr': 79947, 'wr_bytes': 208159115264, 'compress_bytes_used': 0, 'compress_under_bytes': 0, 'stored_raw': 144583966720, 'avail_raw': 112738856448} 2026-02-17T04:58:12.160 DEBUG:tasks.backfill_toofull:compress_ratio: 1.0 2026-02-17T04:58:12.160 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) ^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph-c_325850157d309f65985eea9ec9bb9f1d56726fee/qa/tasks/backfill_toofull.py", line 155, in task assert backfillfull < 0.9 ^^^^^^^^^^^^^^^^^^ AssertionError 2026-02-17T04:58:12.161 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-17T04:58:12.172 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-17T04:58:12.173 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:12.343 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:12.344 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:12.352 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2026-02-17T04:58:12.291223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001276"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79988,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460653+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460653+0000","last_peered":"2026-02-17T04:58:04.460653+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460653+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:12.352 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:12.527 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:12.528 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:12.536 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2026-02-17T04:58:12.291223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001276"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79988,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460653+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460653+0000","last_peered":"2026-02-17T04:58:04.460653+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460653+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:12.537 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:12.714 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:12.714 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:12.724 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2026-02-17T04:58:12.291223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001276"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79988,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460653+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460653+0000","last_peered":"2026-02-17T04:58:04.460653+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460653+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:12.725 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:12.725 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79988, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:04.460653+0000', 'last_change': '2026-02-17T04:58:01.810816+0000', 'last_active': '2026-02-17T04:58:04.460653+0000', 'last_peered': '2026-02-17T04:58:04.460653+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:04.460653+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:12.725 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:12.943 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:12.944 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":259,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":2,"modified":"2026-02-17T04:54:58.264859+0000","services":{}},"progress_events":{}} 2026-02-17T04:58:12.953 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:12.953 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:13.125 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:13.125 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:13.134 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2026-02-17T04:58:12.291223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001276"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79988,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460653+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460653+0000","last_peered":"2026-02-17T04:58:04.460653+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460653+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:16.135 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:16.313 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:16.314 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:16.322 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2026-02-17T04:58:16.291824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.832353"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:16.323 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:16.497 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:16.497 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:16.506 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2026-02-17T04:58:16.291824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.832353"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:16.508 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:16.508 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79989, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:04.460733+0000', 'last_change': '2026-02-17T04:58:01.810816+0000', 'last_active': '2026-02-17T04:58:04.460733+0000', 'last_peered': '2026-02-17T04:58:04.460733+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:04.460733+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:16.509 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:16.727 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:16.727 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":263,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":2,"modified":"2026-02-17T04:54:58.264859+0000","services":{}},"progress_events":{}} 2026-02-17T04:58:16.736 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:16.736 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:16.910 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:16.911 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:16.919 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2026-02-17T04:58:16.291824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.832353"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:19.921 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:20.097 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:20.098 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:20.106 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2026-02-17T04:58:18.292059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:20.107 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:20.284 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:20.284 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:20.293 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2026-02-17T04:58:18.292059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:20.294 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:20.294 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79989, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:04.460733+0000', 'last_change': '2026-02-17T04:58:01.810816+0000', 'last_active': '2026-02-17T04:58:04.460733+0000', 'last_peered': '2026-02-17T04:58:04.460733+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:04.460733+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:20.294 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:20.512 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:20.513 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":267,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":2,"modified":"2026-02-17T04:54:58.264859+0000","services":{}},"progress_events":{}} 2026-02-17T04:58:20.522 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:20.522 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:20.698 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:20.699 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:20.707 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2026-02-17T04:58:20.292298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001532"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:23.709 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:23.887 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:23.888 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:23.897 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2026-02-17T04:58:22.292533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:23.898 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:24.079 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:24.079 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:24.089 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2026-02-17T04:58:22.292533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:24.090 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:24.090 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79989, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:04.460733+0000', 'last_change': '2026-02-17T04:58:01.810816+0000', 'last_active': '2026-02-17T04:58:04.460733+0000', 'last_peered': '2026-02-17T04:58:04.460733+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:04.460733+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:24.090 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:24.307 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:24.307 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":271,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":2,"modified":"2026-02-17T04:54:58.264859+0000","services":{}},"progress_events":{}} 2026-02-17T04:58:24.316 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:24.316 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:24.495 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:24.496 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:24.505 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2026-02-17T04:58:24.292790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001567"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084295,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:27.506 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:27.681 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:27.681 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:27.687 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2026-02-17T04:58:26.293043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001540"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084295,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:27.688 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:27.867 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:27.867 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:27.876 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2026-02-17T04:58:26.293043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001540"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084295,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:27.876 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:27.876 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79989, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:04.460733+0000', 'last_change': '2026-02-17T04:58:01.810816+0000', 'last_active': '2026-02-17T04:58:04.460733+0000', 'last_peered': '2026-02-17T04:58:04.460733+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:04.460733+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:27.877 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:28.092 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:28.092 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":275,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":2,"modified":"2026-02-17T04:54:58.264859+0000","services":{}},"progress_events":{}} 2026-02-17T04:58:28.102 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:28.102 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:28.275 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:28.275 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:28.284 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2026-02-17T04:58:26.293043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001540"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084295,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:31.285 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:31.463 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:31.463 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:31.472 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2026-02-17T04:58:30.293494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001434"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:31.472 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:31.646 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:31.646 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:31.654 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2026-02-17T04:58:30.293494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001434"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:31.654 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:31.654 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79989, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:04.460733+0000', 'last_change': '2026-02-17T04:58:01.810816+0000', 'last_active': '2026-02-17T04:58:04.460733+0000', 'last_peered': '2026-02-17T04:58:04.460733+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:04.460733+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:31.655 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:31.868 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:31.868 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":278,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":2,"modified":"2026-02-17T04:54:58.264859+0000","services":{}},"progress_events":{}} 2026-02-17T04:58:31.877 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:31.877 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:32.052 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:32.052 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:32.061 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2026-02-17T04:58:30.293494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001434"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79989,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:04.460733+0000","last_change":"2026-02-17T04:58:01.810816+0000","last_active":"2026-02-17T04:58:04.460733+0000","last_peered":"2026-02-17T04:58:04.460733+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:04.460733+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:35.063 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:35.237 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:35.238 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:35.246 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2026-02-17T04:58:34.293973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001439"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:35.247 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:35.420 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:35.420 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:35.428 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2026-02-17T04:58:34.293973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001439"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:35.428 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:35.428 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79993, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:31.811363+0000', 'last_change': '2026-02-17T04:58:31.811363+0000', 'last_active': '2026-02-17T04:58:31.811363+0000', 'last_peered': '2026-02-17T04:58:31.811363+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:31.811363+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:35.428 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:35.655 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:35.655 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":282,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":2,"modified":"2026-02-17T04:54:58.264859+0000","services":{}},"progress_events":{}} 2026-02-17T04:58:35.664 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:35.665 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:35.837 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:35.838 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:35.846 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2026-02-17T04:58:34.293973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001439"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:38.847 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:39.023 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:39.023 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:39.032 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2026-02-17T04:58:38.294488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001444"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:39.033 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:39.207 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:39.207 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:39.216 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2026-02-17T04:58:38.294488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001444"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:39.217 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:39.217 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79993, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:31.811363+0000', 'last_change': '2026-02-17T04:58:31.811363+0000', 'last_active': '2026-02-17T04:58:31.811363+0000', 'last_peered': '2026-02-17T04:58:31.811363+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:31.811363+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:39.217 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:39.485 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:58:39.484+0000 7fc514990640 0 --2- 10.20.193.100:0/2086663023 >> [v2:10.20.193.100:3302/0,v1:10.20.193.100:6791/0] conn(0x7fc51005bb80 0x7fc51005bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=6205840863030763753 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T04:58:39.638 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:39.638 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":286,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":2,"modified":"2026-02-17T04:54:58.264859+0000","services":{}},"progress_events":{}} 2026-02-17T04:58:39.647 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:39.647 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:39.822 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:39.822 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:39.830 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2026-02-17T04:58:38.294488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001444"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:42.832 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:43.011 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:43.011 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:43.020 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2026-02-17T04:58:42.295010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001515"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:43.021 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:43.289 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:58:43.288+0000 7fa436ffd640 0 --2- 10.20.193.100:0/1662111732 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7fa438083800 0x7fa438083bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=234955421806583707 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T04:58:43.398 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:43.398 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:43.407 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2026-02-17T04:58:42.295010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001515"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:43.408 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:43.408 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79993, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:31.811363+0000', 'last_change': '2026-02-17T04:58:31.811363+0000', 'last_active': '2026-02-17T04:58:31.811363+0000', 'last_peered': '2026-02-17T04:58:31.811363+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:31.811363+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:43.408 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:43.679 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:58:43.677+0000 7fc54b7fe640 0 --2- 10.20.193.100:0/987668647 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7fc54c080090 0x7fc54c07a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T04:58:43.830 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:43.830 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":290,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":2,"modified":"2026-02-17T04:54:58.264859+0000","services":{}},"progress_events":{}} 2026-02-17T04:58:43.839 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:43.839 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:44.016 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:44.016 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:44.024 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2026-02-17T04:58:42.295010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001515"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:47.026 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:47.201 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:47.201 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:47.210 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2026-02-17T04:58:46.295514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084299,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:47.210 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:47.384 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:47.384 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:47.393 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2026-02-17T04:58:46.295514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084299,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:47.394 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:47.394 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79993, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:31.811363+0000', 'last_change': '2026-02-17T04:58:31.811363+0000', 'last_active': '2026-02-17T04:58:31.811363+0000', 'last_peered': '2026-02-17T04:58:31.811363+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:31.811363+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:47.395 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:47.610 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:47.610 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":294,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":3,"modified":"2026-02-17T04:58:44.295288+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:58:47.619 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:47.620 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:47.793 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:47.794 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:47.801 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2026-02-17T04:58:46.295514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084299,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:50.803 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:50.981 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:50.981 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:50.990 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2026-02-17T04:58:50.296073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001585"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084300,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:50.991 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:51.163 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:51.164 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:51.172 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2026-02-17T04:58:50.296073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001585"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084300,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:51.173 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:51.173 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79993, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:31.811363+0000', 'last_change': '2026-02-17T04:58:31.811363+0000', 'last_active': '2026-02-17T04:58:31.811363+0000', 'last_peered': '2026-02-17T04:58:31.811363+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:31.811363+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:51.174 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:51.388 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:51.388 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":298,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":3,"modified":"2026-02-17T04:58:44.295288+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:58:51.397 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:51.397 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:51.572 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:51.573 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:51.581 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2026-02-17T04:58:50.296073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001585"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084300,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:54.583 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:54.756 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:54.756 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:54.765 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2026-02-17T04:58:54.296635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001624"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:54.766 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:54.939 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:54.939 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:54.947 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2026-02-17T04:58:54.296635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001624"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:54.948 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:54.948 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79993, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:31.811363+0000', 'last_change': '2026-02-17T04:58:31.811363+0000', 'last_active': '2026-02-17T04:58:31.811363+0000', 'last_peered': '2026-02-17T04:58:31.811363+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:31.811363+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:54.949 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:55.171 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:55.171 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":302,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":3,"modified":"2026-02-17T04:58:44.295288+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:58:55.181 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:55.181 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:56.251 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:56.251 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:56.263 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2026-02-17T04:58:54.296635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001624"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:59.266 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:59.446 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:59.446 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:59.455 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2026-02-17T04:58:58.297107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001592"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:59.456 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:58:59.636 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:59.636 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:58:59.645 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2026-02-17T04:58:58.297107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001592"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:58:59.646 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:58:59.646 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79993, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:31.811363+0000', 'last_change': '2026-02-17T04:58:31.811363+0000', 'last_active': '2026-02-17T04:58:31.811363+0000', 'last_peered': '2026-02-17T04:58:31.811363+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:31.811363+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:58:59.646 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:58:59.866 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:58:59.866 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded","count":1},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":306,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":3,"modified":"2026-02-17T04:58:44.295288+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:58:59.875 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:58:59.875 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:00.049 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:00.050 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:00.058 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2026-02-17T04:58:58.297107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001592"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:03.060 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:03.240 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:03.240 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:03.250 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2026-02-17T04:59:02.297700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:03.251 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:03.428 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:03.429 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:03.437 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2026-02-17T04:59:02.297700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:03.438 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:03.438 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79993, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:58:31.811363+0000', 'last_change': '2026-02-17T04:58:31.811363+0000', 'last_active': '2026-02-17T04:58:31.811363+0000', 'last_peered': '2026-02-17T04:58:31.811363+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:58:31.811363+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:03.439 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:03.652 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:03.653 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":310,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":4,"modified":"2026-02-17T04:59:00.297462+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:03.662 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:03.663 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:03.839 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:03.840 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:03.848 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2026-02-17T04:59:02.297700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79993,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:58:31.811363+0000","last_change":"2026-02-17T04:58:31.811363+0000","last_active":"2026-02-17T04:58:31.811363+0000","last_peered":"2026-02-17T04:58:31.811363+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:58:31.811363+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:06.850 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:07.124 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:07.124 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:07.133 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2026-02-17T04:59:06.298206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001570"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:07.134 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:07.311 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:07.311 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:07.321 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2026-02-17T04:59:06.298206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001570"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:07.322 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:07.322 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79997, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:01.811938+0000', 'last_change': '2026-02-17T04:59:01.811938+0000', 'last_active': '2026-02-17T04:59:01.811938+0000', 'last_peered': '2026-02-17T04:59:01.811938+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:01.811938+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:07.322 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:07.539 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:07.540 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":314,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":4,"modified":"2026-02-17T04:59:00.297462+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:07.549 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:07.549 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:07.723 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:07.723 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:07.732 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2026-02-17T04:59:06.298206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001570"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:10.733 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:10.913 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:10.914 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:10.922 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2026-02-17T04:59:10.298736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084304,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:10.923 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:11.099 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:11.100 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:11.108 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2026-02-17T04:59:10.298736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084304,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:11.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:11.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79997, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:01.811938+0000', 'last_change': '2026-02-17T04:59:01.811938+0000', 'last_active': '2026-02-17T04:59:01.811938+0000', 'last_peered': '2026-02-17T04:59:01.811938+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:01.811938+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:11.110 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:11.323 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:11.323 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":318,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":4,"modified":"2026-02-17T04:59:00.297462+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:11.332 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:11.332 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:11.508 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:11.509 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:11.517 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2026-02-17T04:59:10.298736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084304,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:14.519 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:14.696 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:14.697 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:14.705 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2026-02-17T04:59:14.299301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001600"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:14.706 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:14.886 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:14.886 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:14.895 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2026-02-17T04:59:14.299301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001600"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:14.896 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:14.896 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79997, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:01.811938+0000', 'last_change': '2026-02-17T04:59:01.811938+0000', 'last_active': '2026-02-17T04:59:01.811938+0000', 'last_peered': '2026-02-17T04:59:01.811938+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:01.811938+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:14.897 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:15.114 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:15.114 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":322,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:15.123 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:15.124 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:15.299 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:15.299 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:15.308 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2026-02-17T04:59:14.299301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001600"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:18.310 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:18.485 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:18.486 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:18.494 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2026-02-17T04:59:18.299819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001613"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:18.495 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:18.670 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:18.670 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:18.679 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2026-02-17T04:59:18.299819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001613"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:18.680 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:18.680 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79997, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:01.811938+0000', 'last_change': '2026-02-17T04:59:01.811938+0000', 'last_active': '2026-02-17T04:59:01.811938+0000', 'last_peered': '2026-02-17T04:59:01.811938+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:01.811938+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:18.681 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:18.897 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:18.897 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":325,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:18.905 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:18.906 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:19.085 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:19.086 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:19.095 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2026-02-17T04:59:18.299819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001613"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:22.097 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:22.274 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:22.275 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:22.282 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2026-02-17T04:59:20.300045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:22.284 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:22.458 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:22.458 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:22.467 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2026-02-17T04:59:22.300380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001643"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:22.468 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:22.468 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79997, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:01.811938+0000', 'last_change': '2026-02-17T04:59:01.811938+0000', 'last_active': '2026-02-17T04:59:01.811938+0000', 'last_peered': '2026-02-17T04:59:01.811938+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:01.811938+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:22.468 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:22.692 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:22.692 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":329,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:22.702 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:22.702 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:22.875 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:22.876 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:22.884 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2026-02-17T04:59:22.300380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001643"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:25.886 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:26.062 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:26.062 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:26.070 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2026-02-17T04:59:24.300627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001608"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:26.071 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:26.246 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:26.247 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:26.256 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2026-02-17T04:59:24.300627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001608"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:26.257 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:26.257 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79997, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:01.811938+0000', 'last_change': '2026-02-17T04:59:01.811938+0000', 'last_active': '2026-02-17T04:59:01.811938+0000', 'last_peered': '2026-02-17T04:59:01.811938+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:01.811938+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:26.257 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:26.471 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:26.472 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":333,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:26.481 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:26.481 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:26.753 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:26.754 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:26.765 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2026-02-17T04:59:26.300918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001616"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:29.766 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:29.944 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:29.945 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:29.953 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2026-02-17T04:59:28.301145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001581"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:29.954 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:30.224 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:59:30.223+0000 7ff6d27fc640 0 --2- 10.20.193.100:0/3822528201 >> [v2:10.20.193.100:3300/0,v1:10.20.193.100:6789/0] conn(0x7ff6d407f1c0 0x7ff6d4123560 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T04:59:30.331 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:30.332 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:30.340 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2026-02-17T04:59:30.301439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001619"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:30.341 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:30.342 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 79997, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:01.811938+0000', 'last_change': '2026-02-17T04:59:01.811938+0000', 'last_active': '2026-02-17T04:59:01.811938+0000', 'last_peered': '2026-02-17T04:59:01.811938+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:01.811938+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:30.342 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:30.562 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:30.562 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":337,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:30.571 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:30.572 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:30.842 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T04:59:30.841+0000 7fc9fffff640 0 --2- 10.20.193.100:0/1696232201 >> v2:10.20.193.100:3300/0 conn(0x7fca08156150 0x7fca08176530 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T04:59:30.948 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:30.949 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:30.957 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2026-02-17T04:59:30.301439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001619"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:33.959 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:34.136 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:34.136 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:34.145 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2026-02-17T04:59:32.301701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001656"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":79997,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:01.811938+0000","last_change":"2026-02-17T04:59:01.811938+0000","last_active":"2026-02-17T04:59:01.811938+0000","last_peered":"2026-02-17T04:59:01.811938+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:01.811938+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:34.146 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:34.320 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:34.321 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:34.329 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2026-02-17T04:59:34.302012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084309,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:34.330 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:34.330 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80001, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:31.812609+0000', 'last_change': '2026-02-17T04:59:31.812609+0000', 'last_active': '2026-02-17T04:59:31.812609+0000', 'last_peered': '2026-02-17T04:59:31.812609+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:31.812609+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:34.331 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:34.545 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:34.546 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":341,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:34.555 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:34.555 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:34.736 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:34.737 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:34.745 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2026-02-17T04:59:34.302012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084309,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:37.747 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:37.924 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:37.924 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:37.933 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2026-02-17T04:59:36.302280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001653"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084309,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:37.934 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:38.109 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:38.110 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:38.119 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2026-02-17T04:59:36.302280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001653"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084309,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:38.120 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:38.120 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80001, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:31.812609+0000', 'last_change': '2026-02-17T04:59:31.812609+0000', 'last_active': '2026-02-17T04:59:31.812609+0000', 'last_peered': '2026-02-17T04:59:31.812609+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:31.812609+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:38.120 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:38.337 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:38.337 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":345,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:38.346 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:38.346 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:38.523 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:38.524 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:38.532 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2026-02-17T04:59:38.302512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001594"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084309,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:41.535 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:41.713 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:41.713 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:41.722 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2026-02-17T04:59:40.302738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084310,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:41.723 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:41.897 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:41.897 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:41.905 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2026-02-17T04:59:40.302738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084310,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:41.906 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:41.906 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80001, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:31.812609+0000', 'last_change': '2026-02-17T04:59:31.812609+0000', 'last_active': '2026-02-17T04:59:31.812609+0000', 'last_peered': '2026-02-17T04:59:31.812609+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:31.812609+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:41.907 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:42.123 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:42.123 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":349,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:42.132 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:42.132 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:42.308 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:42.308 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:42.317 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2026-02-17T04:59:42.302954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001514"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084310,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:45.318 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:45.500 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:45.500 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:45.508 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2026-02-17T04:59:44.303191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001490"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:45.509 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:45.686 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:45.686 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:45.695 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2026-02-17T04:59:44.303191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001490"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:45.696 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:45.696 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80001, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:31.812609+0000', 'last_change': '2026-02-17T04:59:31.812609+0000', 'last_active': '2026-02-17T04:59:31.812609+0000', 'last_peered': '2026-02-17T04:59:31.812609+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:31.812609+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:45.697 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:45.919 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:45.920 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":352,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:45.929 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:45.929 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:46.104 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:46.105 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:46.113 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2026-02-17T04:59:44.303191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001490"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:49.115 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:49.289 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:49.289 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:49.298 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2026-02-17T04:59:48.303768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001487"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:49.299 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:49.477 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:49.477 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:49.486 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2026-02-17T04:59:48.303768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001487"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:49.487 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:49.487 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80001, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:31.812609+0000', 'last_change': '2026-02-17T04:59:31.812609+0000', 'last_active': '2026-02-17T04:59:31.812609+0000', 'last_peered': '2026-02-17T04:59:31.812609+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:31.812609+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:49.488 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:49.702 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:49.702 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":356,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:49.711 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:49.712 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:49.890 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:49.890 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:49.899 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2026-02-17T04:59:48.303768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001487"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:52.901 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:53.078 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:53.078 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:53.087 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2026-02-17T04:59:52.304232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084312,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:53.088 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:53.359 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:53.360 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:53.369 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2026-02-17T04:59:52.304232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084312,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:53.370 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:53.370 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80001, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:31.812609+0000', 'last_change': '2026-02-17T04:59:31.812609+0000', 'last_active': '2026-02-17T04:59:31.812609+0000', 'last_peered': '2026-02-17T04:59:31.812609+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:31.812609+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:53.370 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:53.589 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:53.589 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":360,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:53.599 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:53.600 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:53.777 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:53.777 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:53.786 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2026-02-17T04:59:52.304232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084312,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:56.788 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:56.963 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:56.964 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:56.972 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2026-02-17T04:59:56.304692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001500"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084313,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:56.973 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:57.154 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:57.154 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:57.164 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2026-02-17T04:59:56.304692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001500"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084313,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T04:59:57.165 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T04:59:57.166 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80001, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:31.812609+0000', 'last_change': '2026-02-17T04:59:31.812609+0000', 'last_active': '2026-02-17T04:59:31.812609+0000', 'last_peered': '2026-02-17T04:59:31.812609+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:31.812609+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T04:59:57.166 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T04:59:57.383 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:57.383 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":364,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T04:59:57.393 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T04:59:57.393 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T04:59:57.569 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T04:59:57.569 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T04:59:57.577 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2026-02-17T04:59:56.304692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001500"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460037+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460037+0000","last_peered":"2026-02-17T04:58:04.460037+0000","last_clean":"2026-02-17T04:58:04.460037+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460037+0000","last_undegraded":"2026-02-17T04:58:04.460037+0000","last_fullsized":"2026-02-17T04:58:04.460037+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084313,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:00.579 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:00.758 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:00.759 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:00.767 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2026-02-17T05:00:00.305071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001303"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084314,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:00.769 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:00.949 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:00.950 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:00.959 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2026-02-17T05:00:00.305071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001303"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084314,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:00.960 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:00.960 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80001, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T04:59:31.812609+0000', 'last_change': '2026-02-17T04:59:31.812609+0000', 'last_active': '2026-02-17T04:59:31.812609+0000', 'last_peered': '2026-02-17T04:59:31.812609+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T04:59:31.812609+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:00.960 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:01.183 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:01.183 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":368,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:01.193 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:01.193 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:01.368 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:01.368 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:01.376 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2026-02-17T05:00:00.305071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001303"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80001,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T04:59:31.812609+0000","last_change":"2026-02-17T04:59:31.812609+0000","last_active":"2026-02-17T04:59:31.812609+0000","last_peered":"2026-02-17T04:59:31.812609+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T04:59:31.812609+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084314,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:04.379 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:04.560 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:04.560 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:04.570 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2026-02-17T05:00:04.305542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001309"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80005,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813192+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813192+0000","last_peered":"2026-02-17T05:00:01.813192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084315,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:04.571 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:04.748 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:04.748 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:04.758 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2026-02-17T05:00:04.305542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001309"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80005,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813192+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813192+0000","last_peered":"2026-02-17T05:00:01.813192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084315,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:04.759 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:04.759 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80005, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:01.813192+0000', 'last_change': '2026-02-17T05:00:01.813192+0000', 'last_active': '2026-02-17T05:00:01.813192+0000', 'last_peered': '2026-02-17T05:00:01.813192+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:01.813192+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:04.759 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:04.978 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:04.978 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":371,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:04.988 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:04.988 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:05.161 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:05.161 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:05.169 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2026-02-17T05:00:04.305542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001309"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80005,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813192+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813192+0000","last_peered":"2026-02-17T05:00:01.813192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084315,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:08.171 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:08.351 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:08.351 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:08.360 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2026-02-17T05:00:08.306076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001384"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80005,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813192+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813192+0000","last_peered":"2026-02-17T05:00:01.813192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084315,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:08.361 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:08.539 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:08.539 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:08.547 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2026-02-17T05:00:08.306076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001384"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80005,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813192+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813192+0000","last_peered":"2026-02-17T05:00:01.813192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084315,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:08.548 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:08.549 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80005, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:01.813192+0000', 'last_change': '2026-02-17T05:00:01.813192+0000', 'last_active': '2026-02-17T05:00:01.813192+0000', 'last_peered': '2026-02-17T05:00:01.813192+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:01.813192+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:08.549 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:08.768 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:08.768 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":375,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:08.778 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:08.778 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:08.952 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:08.953 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:08.961 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2026-02-17T05:00:08.306076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001384"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80005,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813192+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813192+0000","last_peered":"2026-02-17T05:00:01.813192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084315,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:11.963 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:12.141 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:12.141 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:12.150 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2026-02-17T05:00:10.306294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001416"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80005,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813192+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813192+0000","last_peered":"2026-02-17T05:00:01.813192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:12.151 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:12.329 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:12.330 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:12.338 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2026-02-17T05:00:12.306541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001470"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80005,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813192+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813192+0000","last_peered":"2026-02-17T05:00:01.813192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:12.339 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:12.339 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80005, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:01.813192+0000', 'last_change': '2026-02-17T05:00:01.813192+0000', 'last_active': '2026-02-17T05:00:01.813192+0000', 'last_peered': '2026-02-17T05:00:01.813192+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:01.813192+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:12.340 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:12.561 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:12.562 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":379,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T04:59:14.299333+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:12.571 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:12.571 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:12.745 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:12.746 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:12.755 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2026-02-17T05:00:12.306541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001470"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80005,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813192+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813192+0000","last_peered":"2026-02-17T05:00:01.813192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:15.757 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:15.936 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:15.936 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:15.944 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2026-02-17T05:00:14.306744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001418"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084317,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:15.946 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:16.121 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:16.122 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:16.130 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2026-02-17T05:00:14.306744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001418"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084317,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:16.131 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:16.131 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80006, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:01.813225+0000', 'last_change': '2026-02-17T05:00:01.813192+0000', 'last_active': '2026-02-17T05:00:01.813225+0000', 'last_peered': '2026-02-17T05:00:01.813225+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:01.813225+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:16.132 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:16.351 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:16.352 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":383,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-17T05:00:14.306802+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:16.361 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:16.362 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:16.537 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:16.537 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:16.546 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2026-02-17T05:00:16.307055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084317,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:19.548 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:19.725 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:19.725 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:19.734 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2026-02-17T05:00:18.307311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084317,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:19.735 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:19.915 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:19.915 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:19.924 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2026-02-17T05:00:18.307311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084317,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:19.925 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:19.925 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80006, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:01.813225+0000', 'last_change': '2026-02-17T05:00:01.813192+0000', 'last_active': '2026-02-17T05:00:01.813225+0000', 'last_peered': '2026-02-17T05:00:01.813225+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:01.813225+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:19.926 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:20.141 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:20.141 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":387,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-17T05:00:14.306802+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:20.150 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:20.150 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:20.331 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:20.332 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:20.340 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2026-02-17T05:00:20.307601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084318,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:23.341 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:23.518 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:23.518 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:23.527 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2026-02-17T05:00:22.307855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084318,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:23.528 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:23.703 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:23.704 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:23.713 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2026-02-17T05:00:22.307855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084318,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:23.714 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:23.714 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80006, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:01.813225+0000', 'last_change': '2026-02-17T05:00:01.813192+0000', 'last_active': '2026-02-17T05:00:01.813225+0000', 'last_peered': '2026-02-17T05:00:01.813225+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:01.813225+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:23.715 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:23.940 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:23.940 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":390,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-17T05:00:14.306802+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:23.949 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:23.949 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:24.126 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:24.126 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:24.135 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2026-02-17T05:00:22.307855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084318,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:27.136 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:27.312 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:27.313 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:27.321 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2026-02-17T05:00:26.308387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001643"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:27.322 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:27.498 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:27.498 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:27.507 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2026-02-17T05:00:26.308387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001643"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:27.508 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:27.508 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80006, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:01.813225+0000', 'last_change': '2026-02-17T05:00:01.813192+0000', 'last_active': '2026-02-17T05:00:01.813225+0000', 'last_peered': '2026-02-17T05:00:01.813225+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:01.813225+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:27.508 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:27.728 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:27.729 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":394,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-17T05:00:14.306802+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:27.737 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:27.738 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:27.915 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:27.915 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:27.924 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2026-02-17T05:00:26.308387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001643"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:30.926 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:31.106 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:31.106 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:31.115 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2026-02-17T05:00:30.308856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001544"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:31.116 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:31.296 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:31.296 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:31.305 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2026-02-17T05:00:30.308856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001544"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:31.306 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:31.306 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80006, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:01.813225+0000', 'last_change': '2026-02-17T05:00:01.813192+0000', 'last_active': '2026-02-17T05:00:01.813225+0000', 'last_peered': '2026-02-17T05:00:01.813225+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:01.813225+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:31.307 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:31.524 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:31.524 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":398,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-17T05:00:14.306802+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:31.534 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:31.534 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:31.710 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:31.710 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:31.720 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2026-02-17T05:00:30.308856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001544"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80006,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:01.813225+0000","last_change":"2026-02-17T05:00:01.813192+0000","last_active":"2026-02-17T05:00:01.813225+0000","last_peered":"2026-02-17T05:00:01.813225+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:01.813225+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:34.722 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:35.796 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:35.796 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:35.806 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2026-02-17T05:00:34.309323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001467"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:35.807 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:36.078 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:36.079 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:36.087 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2026-02-17T05:00:34.309323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001467"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:36.088 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:36.088 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80010, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:31.813859+0000', 'last_change': '2026-02-17T05:00:31.813859+0000', 'last_active': '2026-02-17T05:00:31.813859+0000', 'last_peered': '2026-02-17T05:00:31.813859+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:31.813859+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:36.089 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:36.304 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:36.304 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":403,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-17T05:00:14.306802+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:36.313 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:36.313 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:36.489 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:36.489 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:36.498 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2026-02-17T05:00:36.309586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001453"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:39.500 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:39.677 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:39.677 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:39.685 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2026-02-17T05:00:38.309829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001441"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:39.686 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:39.860 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:39.860 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:39.869 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2026-02-17T05:00:38.309829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001441"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:39.870 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:39.870 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80010, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:31.813859+0000', 'last_change': '2026-02-17T05:00:31.813859+0000', 'last_active': '2026-02-17T05:00:31.813859+0000', 'last_peered': '2026-02-17T05:00:31.813859+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:31.813859+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:39.870 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:40.084 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:40.084 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":407,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-17T05:00:14.306802+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:40.093 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:40.093 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:40.268 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:40.269 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:40.277 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2026-02-17T05:00:38.309829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001441"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:43.280 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:43.460 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:43.461 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:43.469 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2026-02-17T05:00:42.310369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:43.470 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:43.643 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:43.644 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:43.651 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2026-02-17T05:00:42.310369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:43.652 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:43.653 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80010, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:31.813859+0000', 'last_change': '2026-02-17T05:00:31.813859+0000', 'last_active': '2026-02-17T05:00:31.813859+0000', 'last_peered': '2026-02-17T05:00:31.813859+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:31.813859+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:43.653 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:43.874 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:43.875 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":410,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-17T05:00:14.306802+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:43.883 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:43.884 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:44.057 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:44.058 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:44.067 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2026-02-17T05:00:42.310369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:47.068 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:47.248 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:47.248 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:47.258 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2026-02-17T05:00:46.310854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001531"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:47.259 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:47.434 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:47.435 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:47.443 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2026-02-17T05:00:46.310854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001531"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:47.444 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:47.445 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80010, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:31.813859+0000', 'last_change': '2026-02-17T05:00:31.813859+0000', 'last_active': '2026-02-17T05:00:31.813859+0000', 'last_peered': '2026-02-17T05:00:31.813859+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:31.813859+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:47.445 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:47.661 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:47.661 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":414,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-17T05:00:14.306802+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:47.670 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:47.670 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:47.846 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:47.846 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:47.855 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2026-02-17T05:00:46.310854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001531"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:50.856 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:51.035 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:51.035 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:51.044 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2026-02-17T05:00:50.311364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001534"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:51.045 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:51.218 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:51.218 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:51.227 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2026-02-17T05:00:50.311364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001534"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:51.228 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:51.228 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80010, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:31.813859+0000', 'last_change': '2026-02-17T05:00:31.813859+0000', 'last_active': '2026-02-17T05:00:31.813859+0000', 'last_peered': '2026-02-17T05:00:31.813859+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:31.813859+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:51.229 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:51.446 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:51.446 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":418,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-17T05:00:14.306802+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:51.456 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:51.456 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:51.629 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:51.629 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:51.639 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2026-02-17T05:00:50.311364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001534"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:54.640 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:54.816 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:54.816 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:54.825 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2026-02-17T05:00:54.311916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001546"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:54.825 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:54.998 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:54.999 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:55.008 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2026-02-17T05:00:54.311916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001546"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:55.009 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:55.009 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80010, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:31.813859+0000', 'last_change': '2026-02-17T05:00:31.813859+0000', 'last_active': '2026-02-17T05:00:31.813859+0000', 'last_peered': '2026-02-17T05:00:31.813859+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:31.813859+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:55.009 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:55.224 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:55.225 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":422,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-02-17T05:00:14.306802+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:55.233 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:55.234 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:55.406 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:55.406 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:55.415 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2026-02-17T05:00:54.311916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001546"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:58.416 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:58.590 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:58.591 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:58.599 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2026-02-17T05:00:58.312326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001471"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:58.600 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:58.774 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:58.775 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:58.784 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2026-02-17T05:00:58.312326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001471"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:00:58.785 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:00:58.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80010, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:31.813859+0000', 'last_change': '2026-02-17T05:00:31.813859+0000', 'last_active': '2026-02-17T05:00:31.813859+0000', 'last_peered': '2026-02-17T05:00:31.813859+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:31.813859+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:00:58.785 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:00:59.003 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:59.003 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":425,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-17T05:00:58.312353+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:00:59.012 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:00:59.012 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:00:59.188 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:00:59.188 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:00:59.197 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2026-02-17T05:00:58.312326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001471"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:02.199 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:02.379 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:02.379 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:02.388 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2026-02-17T05:01:02.312885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001521"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:02.389 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:02.563 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:02.564 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:02.572 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2026-02-17T05:01:02.312885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001521"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:02.573 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:02.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80010, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:00:31.813859+0000', 'last_change': '2026-02-17T05:00:31.813859+0000', 'last_active': '2026-02-17T05:00:31.813859+0000', 'last_peered': '2026-02-17T05:00:31.813859+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:00:31.813859+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:02.574 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:02.790 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:02.790 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":429,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-17T05:00:58.312353+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:01:02.800 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:02.800 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:02.979 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:02.979 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:02.988 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2026-02-17T05:01:02.312885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001521"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80010,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:00:31.813859+0000","last_change":"2026-02-17T05:00:31.813859+0000","last_active":"2026-02-17T05:00:31.813859+0000","last_peered":"2026-02-17T05:00:31.813859+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:00:31.813859+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:05.990 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:06.165 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:06.165 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:06.173 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2026-02-17T05:01:04.313124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001496"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084327,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:06.175 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:06.354 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:06.354 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:06.363 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2026-02-17T05:01:06.313426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001510"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084327,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:06.364 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:06.364 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80014, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:01.814450+0000', 'last_change': '2026-02-17T05:01:01.814450+0000', 'last_active': '2026-02-17T05:01:01.814450+0000', 'last_peered': '2026-02-17T05:01:01.814450+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:01.814450+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:06.364 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:06.581 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:06.581 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":433,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-17T05:00:58.312353+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:01:06.591 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:06.591 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:06.768 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:06.769 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:06.778 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2026-02-17T05:01:06.313426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001510"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084327,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:09.780 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:09.958 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:09.959 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:09.967 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2026-02-17T05:01:08.313677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084327,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:09.968 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:10.147 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:10.147 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:10.157 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2026-02-17T05:01:08.313677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084327,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:10.158 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:10.158 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80014, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:01.814450+0000', 'last_change': '2026-02-17T05:01:01.814450+0000', 'last_active': '2026-02-17T05:01:01.814450+0000', 'last_peered': '2026-02-17T05:01:01.814450+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:01.814450+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:10.158 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:10.383 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:10.383 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":437,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-17T05:00:58.312353+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:01:10.393 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:10.393 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:10.569 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:10.570 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:10.579 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2026-02-17T05:01:10.313910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001583"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084328,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:13.581 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:13.757 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:13.758 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:13.769 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2026-02-17T05:01:12.314190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001610"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084328,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:13.770 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:13.949 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:13.949 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:13.958 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2026-02-17T05:01:12.314190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001610"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084328,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705756,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:13.959 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:13.959 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80014, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:01.814450+0000', 'last_change': '2026-02-17T05:01:01.814450+0000', 'last_active': '2026-02-17T05:01:01.814450+0000', 'last_peered': '2026-02-17T05:01:01.814450+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:01.814450+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:13.959 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:14.178 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:14.178 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":441,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-17T05:00:58.312353+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:01:14.187 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:14.187 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:14.366 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:14.367 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:14.376 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2026-02-17T05:01:14.314511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084329,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:17.378 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:17.554 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:17.554 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:17.563 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2026-02-17T05:01:16.314755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001630"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084329,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:17.564 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:17.742 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:17.742 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:17.750 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2026-02-17T05:01:16.314755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001630"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084329,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:17.752 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:17.752 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80014, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:01.814450+0000', 'last_change': '2026-02-17T05:01:01.814450+0000', 'last_active': '2026-02-17T05:01:01.814450+0000', 'last_peered': '2026-02-17T05:01:01.814450+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:01.814450+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:17.752 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:17.972 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:17.972 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":444,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":7,"modified":"2026-02-17T05:00:58.312353+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:01:17.981 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:17.981 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:18.156 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:18.157 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:18.166 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2026-02-17T05:01:16.314755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001630"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084329,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:21.167 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:21.345 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:21.346 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:21.354 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2026-02-17T05:01:20.315198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001521"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084330,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:21.355 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:21.530 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:21.531 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:21.539 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2026-02-17T05:01:20.315198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001521"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084330,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:21.540 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:21.540 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80014, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:01.814450+0000', 'last_change': '2026-02-17T05:01:01.814450+0000', 'last_active': '2026-02-17T05:01:01.814450+0000', 'last_peered': '2026-02-17T05:01:01.814450+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:01.814450+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:21.540 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:21.760 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:21.760 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":448,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:01:21.772 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:21.772 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:21.948 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:21.949 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:21.958 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2026-02-17T05:01:20.315198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001521"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084330,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:24.960 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:25.135 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:25.136 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:25.142 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2026-02-17T05:01:24.315761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001570"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:25.143 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:25.324 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:25.324 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:25.333 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2026-02-17T05:01:24.315761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001570"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:25.334 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:25.334 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80014, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:01.814450+0000', 'last_change': '2026-02-17T05:01:01.814450+0000', 'last_active': '2026-02-17T05:01:01.814450+0000', 'last_peered': '2026-02-17T05:01:01.814450+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:01.814450+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:25.335 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:25.552 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:25.552 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":452,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:01:25.561 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:25.562 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:25.737 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:25.738 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:25.746 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2026-02-17T05:01:24.315761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001570"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:28.748 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:28.922 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:28.922 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:28.930 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2026-02-17T05:01:28.316210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001454"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:28.931 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:29.107 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:29.107 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:29.116 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2026-02-17T05:01:28.316210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001454"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:29.118 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:29.118 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80014, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:01.814450+0000', 'last_change': '2026-02-17T05:01:01.814450+0000', 'last_active': '2026-02-17T05:01:01.814450+0000', 'last_peered': '2026-02-17T05:01:01.814450+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:01.814450+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:29.118 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:29.338 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:29.338 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":456,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:01:29.347 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:29.348 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:29.524 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:29.524 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:29.534 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2026-02-17T05:01:28.316210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001454"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:32.536 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:32.715 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:32.715 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:32.723 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2026-02-17T05:01:32.316689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001490"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:32.724 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:32.899 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:32.900 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:32.908 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2026-02-17T05:01:32.316689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001490"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:32.909 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:32.909 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80014, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:01.814450+0000', 'last_change': '2026-02-17T05:01:01.814450+0000', 'last_active': '2026-02-17T05:01:01.814450+0000', 'last_peered': '2026-02-17T05:01:01.814450+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:01.814450+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:32.910 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:33.132 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:33.132 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":460,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:01:33.141 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:33.141 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:33.319 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:33.319 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:33.328 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2026-02-17T05:01:32.316689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001490"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80014,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:01.814450+0000","last_change":"2026-02-17T05:01:01.814450+0000","last_active":"2026-02-17T05:01:01.814450+0000","last_peered":"2026-02-17T05:01:01.814450+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:01.814450+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:36.329 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:36.505 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:36.506 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:36.514 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2026-02-17T05:01:36.317206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001444"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:36.515 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:36.693 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:36.693 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:36.702 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2026-02-17T05:01:36.317206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001444"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:36.703 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:36.703 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80018, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:31.814950+0000', 'last_change': '2026-02-17T05:01:31.814950+0000', 'last_active': '2026-02-17T05:01:31.814950+0000', 'last_peered': '2026-02-17T05:01:31.814950+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:31.814950+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:36.704 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:36.924 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:36.924 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":463,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:01:36.934 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:36.934 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:37.110 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:37.110 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:37.118 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2026-02-17T05:01:36.317206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001444"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:40.121 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:40.303 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:40.304 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:40.313 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2026-02-17T05:01:38.317432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001423"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:40.314 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:40.490 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:40.491 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:40.499 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2026-02-17T05:01:40.317727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:40.500 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:40.501 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80018, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:31.814950+0000', 'last_change': '2026-02-17T05:01:31.814950+0000', 'last_active': '2026-02-17T05:01:31.814950+0000', 'last_peered': '2026-02-17T05:01:31.814950+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:31.814950+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:40.501 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:40.719 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:40.719 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":467,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:01:40.728 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:40.729 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:40.904 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:40.904 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:40.913 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2026-02-17T05:01:40.317727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:43.915 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:44.092 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:44.093 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:44.102 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2026-02-17T05:01:42.317982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001526"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:44.103 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:44.278 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:44.278 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:44.287 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2026-02-17T05:01:42.317982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001526"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:44.288 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:44.288 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80018, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:31.814950+0000', 'last_change': '2026-02-17T05:01:31.814950+0000', 'last_active': '2026-02-17T05:01:31.814950+0000', 'last_peered': '2026-02-17T05:01:31.814950+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:31.814950+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:44.289 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:44.511 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:44.511 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":471,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:01:44.521 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:44.521 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:44.696 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:44.696 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:44.704 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2026-02-17T05:01:44.318278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:47.705 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:47.884 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:47.884 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:47.893 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2026-02-17T05:01:46.318541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:47.894 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:48.070 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:48.071 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:48.079 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2026-02-17T05:01:46.318541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:48.080 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:48.081 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80018, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:31.814950+0000', 'last_change': '2026-02-17T05:01:31.814950+0000', 'last_active': '2026-02-17T05:01:31.814950+0000', 'last_peered': '2026-02-17T05:01:31.814950+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:31.814950+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:48.081 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:48.293 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:48.293 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":475,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:01:48.301 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:48.301 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:48.475 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:48.475 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:48.483 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2026-02-17T05:01:48.318837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001630"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:51.486 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:51.670 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:51.670 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:51.679 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2026-02-17T05:01:50.319070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001638"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:51.680 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:51.855 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:51.856 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:51.864 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2026-02-17T05:01:50.319070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001638"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:51.865 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:51.866 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80018, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:31.814950+0000', 'last_change': '2026-02-17T05:01:31.814950+0000', 'last_active': '2026-02-17T05:01:31.814950+0000', 'last_peered': '2026-02-17T05:01:31.814950+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:31.814950+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:51.866 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:52.092 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:52.092 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":479,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:01:52.101 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:52.102 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:52.281 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:52.282 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:52.290 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2026-02-17T05:01:50.319070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001638"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:55.292 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:55.467 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:55.467 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:55.476 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2026-02-17T05:01:54.319520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001537"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:55.477 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:55.654 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:55.655 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:55.663 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2026-02-17T05:01:54.319520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001537"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:55.664 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:55.664 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80018, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:31.814950+0000', 'last_change': '2026-02-17T05:01:31.814950+0000', 'last_active': '2026-02-17T05:01:31.814950+0000', 'last_peered': '2026-02-17T05:01:31.814950+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:31.814950+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:55.665 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:55.883 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:55.883 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":482,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:01:55.893 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:55.893 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:56.064 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:56.064 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:56.073 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2026-02-17T05:01:54.319520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001537"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705765,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:59.075 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:59.250 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:59.250 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:59.258 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2026-02-17T05:01:58.320048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:59.260 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:59.437 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:59.437 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:59.446 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2026-02-17T05:01:58.320048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:01:59.447 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:01:59.447 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80018, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:31.814950+0000', 'last_change': '2026-02-17T05:01:31.814950+0000', 'last_active': '2026-02-17T05:01:31.814950+0000', 'last_peered': '2026-02-17T05:01:31.814950+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:31.814950+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:01:59.447 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:01:59.675 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:59.675 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":486,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:01:59.684 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:01:59.685 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:01:59.861 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:01:59.862 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:01:59.870 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2026-02-17T05:01:58.320048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:58:04.460075+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:58:04.460075+0000","last_peered":"2026-02-17T04:58:04.460075+0000","last_clean":"2026-02-17T04:58:04.460075+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:58:04.460075+0000","last_undegraded":"2026-02-17T04:58:04.460075+0000","last_fullsized":"2026-02-17T04:58:04.460075+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:02.872 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:03.057 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:03.058 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:03.067 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2026-02-17T05:02:02.320529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001458"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:03.069 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:03.247 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:03.248 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:03.256 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2026-02-17T05:02:02.320529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001458"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:03.257 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:03.258 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80018, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:01:31.814950+0000', 'last_change': '2026-02-17T05:01:31.814950+0000', 'last_active': '2026-02-17T05:01:31.814950+0000', 'last_peered': '2026-02-17T05:01:31.814950+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:01:31.814950+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:03.258 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:03.477 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:03.477 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":490,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:03.487 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:03.487 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:03.662 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:03.662 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:03.671 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2026-02-17T05:02:02.320529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001458"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80018,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:01:31.814950+0000","last_change":"2026-02-17T05:01:31.814950+0000","last_active":"2026-02-17T05:01:31.814950+0000","last_peered":"2026-02-17T05:01:31.814950+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:01:31.814950+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:06.673 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:06.853 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:06.854 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:06.863 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2026-02-17T05:02:06.321103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001583"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80022,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815425+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815425+0000","last_peered":"2026-02-17T05:02:01.815425+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815425+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:06.864 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:07.040 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:07.040 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:07.049 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2026-02-17T05:02:06.321103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001583"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80022,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815425+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815425+0000","last_peered":"2026-02-17T05:02:01.815425+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815425+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:07.050 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:07.050 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80022, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:01.815425+0000', 'last_change': '2026-02-17T05:02:01.815425+0000', 'last_active': '2026-02-17T05:02:01.815425+0000', 'last_peered': '2026-02-17T05:02:01.815425+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:01.815425+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:07.051 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:07.269 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:07.269 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":494,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:07.278 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:07.279 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:07.456 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:07.457 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:07.466 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2026-02-17T05:02:06.321103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001583"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80022,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815425+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815425+0000","last_peered":"2026-02-17T05:02:01.815425+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815425+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705767,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:10.468 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:10.647 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:10.647 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:10.657 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2026-02-17T05:02:10.321645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001597"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80022,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815425+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815425+0000","last_peered":"2026-02-17T05:02:01.815425+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815425+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084340,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:10.658 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:10.834 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:10.835 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:10.844 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2026-02-17T05:02:10.321645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001597"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80022,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815425+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815425+0000","last_peered":"2026-02-17T05:02:01.815425+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815425+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084340,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:10.845 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:10.845 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80022, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:01.815425+0000', 'last_change': '2026-02-17T05:02:01.815425+0000', 'last_active': '2026-02-17T05:02:01.815425+0000', 'last_peered': '2026-02-17T05:02:01.815425+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:01.815425+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:10.845 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:11.062 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:11.063 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":497,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:11.071 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:11.071 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:11.248 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:11.249 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:11.257 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2026-02-17T05:02:10.321645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001597"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80022,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815425+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815425+0000","last_peered":"2026-02-17T05:02:01.815425+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815425+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084340,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:14.260 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:14.439 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:14.439 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:14.448 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2026-02-17T05:02:14.322211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001681"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:14.448 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:14.623 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:14.623 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:14.631 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2026-02-17T05:02:14.322211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001681"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:14.632 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:14.632 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80023, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:01.815467+0000', 'last_change': '2026-02-17T05:02:01.815425+0000', 'last_active': '2026-02-17T05:02:01.815467+0000', 'last_peered': '2026-02-17T05:02:01.815467+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:01.815467+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:14.633 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:14.849 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:14.849 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":501,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:14.857 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:14.858 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:15.034 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:15.034 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:15.042 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2026-02-17T05:02:14.322211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001681"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:18.044 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:18.221 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:18.221 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:18.230 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2026-02-17T05:02:16.322453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001663"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:18.231 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:18.406 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:18.407 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:18.415 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2026-02-17T05:02:18.322777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001673"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:18.416 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:18.416 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80023, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:01.815467+0000', 'last_change': '2026-02-17T05:02:01.815425+0000', 'last_active': '2026-02-17T05:02:01.815467+0000', 'last_peered': '2026-02-17T05:02:01.815467+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:01.815467+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:18.417 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:18.632 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:18.632 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":505,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T05:01:18.314993+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:18.641 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:18.642 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:18.911 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:02:18.909+0000 7f9cde575640 0 --2- 10.20.193.100:0/656146527 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7f9ce007e8b0 0x7f9ce007ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=15359550798411678715 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:02:19.020 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:19.021 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:19.030 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2026-02-17T05:02:18.322777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001673"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:22.032 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:22.210 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:22.211 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:22.219 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2026-02-17T05:02:20.323053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001716"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:22.220 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:22.399 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:22.400 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:22.408 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2026-02-17T05:02:22.323373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001728"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:22.409 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:22.409 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80023, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:01.815467+0000', 'last_change': '2026-02-17T05:02:01.815425+0000', 'last_active': '2026-02-17T05:02:01.815467+0000', 'last_peered': '2026-02-17T05:02:01.815467+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:01.815467+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:22.410 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:22.628 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:22.628 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":509,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":9,"modified":"2026-02-17T05:02:20.323109+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:02:22.637 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:22.637 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:22.813 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:22.813 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:22.821 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2026-02-17T05:02:22.323373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001728"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:25.822 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:26.004 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:26.005 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:26.014 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2026-02-17T05:02:24.323610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084343,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:26.015 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:26.188 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:26.189 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:26.197 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2026-02-17T05:02:24.323610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084343,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:26.198 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:26.198 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80023, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:01.815467+0000', 'last_change': '2026-02-17T05:02:01.815425+0000', 'last_active': '2026-02-17T05:02:01.815467+0000', 'last_peered': '2026-02-17T05:02:01.815467+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:01.815467+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:26.198 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:26.416 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:26.417 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":513,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:26.426 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:26.427 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:26.601 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:26.601 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:26.609 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2026-02-17T05:02:26.323913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084343,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:29.611 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:29.790 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:29.790 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:29.798 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2026-02-17T05:02:28.324163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084343,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:29.799 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:29.974 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:29.974 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:29.984 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2026-02-17T05:02:28.324163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084343,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:29.985 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:29.985 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80023, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:01.815467+0000', 'last_change': '2026-02-17T05:02:01.815425+0000', 'last_active': '2026-02-17T05:02:01.815467+0000', 'last_peered': '2026-02-17T05:02:01.815467+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:01.815467+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:29.985 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:30.200 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:30.200 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":517,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:30.208 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:30.209 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:30.380 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:30.381 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:30.389 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2026-02-17T05:02:30.324505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001728"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084344,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:33.392 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:33.568 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:33.569 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:33.577 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2026-02-17T05:02:32.324755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084344,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:33.578 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:33.752 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:33.752 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:33.760 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2026-02-17T05:02:32.324755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084344,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:33.761 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:33.761 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80023, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:01.815467+0000', 'last_change': '2026-02-17T05:02:01.815425+0000', 'last_active': '2026-02-17T05:02:01.815467+0000', 'last_peered': '2026-02-17T05:02:01.815467+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:01.815467+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:33.761 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:33.985 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:33.985 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":520,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:33.995 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:33.996 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:34.169 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:34.170 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:34.178 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2026-02-17T05:02:32.324755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80023,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:01.815467+0000","last_change":"2026-02-17T05:02:01.815425+0000","last_active":"2026-02-17T05:02:01.815467+0000","last_peered":"2026-02-17T05:02:01.815467+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:01.815467+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084344,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705772,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:37.180 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:37.354 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:37.355 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:37.362 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2026-02-17T05:02:36.325213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001602"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084345,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:37.362 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:37.537 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:37.537 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:37.546 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2026-02-17T05:02:36.325213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001602"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084345,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:37.546 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:37.547 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80027, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:31.816012+0000', 'last_change': '2026-02-17T05:02:31.816012+0000', 'last_active': '2026-02-17T05:02:31.816012+0000', 'last_peered': '2026-02-17T05:02:31.816012+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:31.816012+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:37.547 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:37.761 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:37.761 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":524,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:37.770 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:37.771 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:38.041 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:02:38.039+0000 7f22ab7fe640 0 --2- 10.20.193.100:0/1729190497 >> [v2:10.20.193.100:3300/0,v1:10.20.193.100:6789/0] conn(0x7f22ac175a60 0x7f22ac178f90 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:02:38.148 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:38.148 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:38.156 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2026-02-17T05:02:36.325213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001602"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084345,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:41.159 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:41.336 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:41.336 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:41.344 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2026-02-17T05:02:40.325752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:41.345 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:41.517 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:41.517 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:41.526 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2026-02-17T05:02:40.325752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:41.527 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:41.527 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80027, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:31.816012+0000', 'last_change': '2026-02-17T05:02:31.816012+0000', 'last_active': '2026-02-17T05:02:31.816012+0000', 'last_peered': '2026-02-17T05:02:31.816012+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:31.816012+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:41.527 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:41.748 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:41.748 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":528,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:41.758 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:41.759 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:41.934 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:41.934 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:41.941 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2026-02-17T05:02:40.325752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705774,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:44.943 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:45.123 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:45.123 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:45.133 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2026-02-17T05:02:44.326138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001383"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084347,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:45.135 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:45.312 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:45.312 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:45.321 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2026-02-17T05:02:44.326138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001383"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084347,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:45.322 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:45.322 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80027, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:31.816012+0000', 'last_change': '2026-02-17T05:02:31.816012+0000', 'last_active': '2026-02-17T05:02:31.816012+0000', 'last_peered': '2026-02-17T05:02:31.816012+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:31.816012+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:45.323 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:45.541 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:45.542 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":532,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:45.552 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:45.552 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:45.727 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:45.728 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:45.736 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2026-02-17T05:02:44.326138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001383"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084347,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:48.738 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:48.925 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:48.926 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:48.935 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2026-02-17T05:02:48.326678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001465"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084347,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:48.936 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:49.108 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:49.108 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:49.117 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2026-02-17T05:02:48.326678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001465"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084347,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:49.118 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:49.119 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80027, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:31.816012+0000', 'last_change': '2026-02-17T05:02:31.816012+0000', 'last_active': '2026-02-17T05:02:31.816012+0000', 'last_peered': '2026-02-17T05:02:31.816012+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:31.816012+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:49.119 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:49.336 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:49.337 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":536,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:49.346 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:49.346 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:49.522 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:49.522 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:49.531 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2026-02-17T05:02:48.326678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001465"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084347,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:52.533 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:52.712 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:52.712 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:52.721 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2026-02-17T05:02:52.327252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001499"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:52.722 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:52.900 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:52.900 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:52.909 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2026-02-17T05:02:52.327252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001499"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:52.910 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:52.910 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80027, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:31.816012+0000', 'last_change': '2026-02-17T05:02:31.816012+0000', 'last_active': '2026-02-17T05:02:31.816012+0000', 'last_peered': '2026-02-17T05:02:31.816012+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:31.816012+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:52.911 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:53.131 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:53.131 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":540,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:53.140 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:53.141 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:53.314 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:53.314 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:53.323 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2026-02-17T05:02:52.327252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001499"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:56.324 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:56.503 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:56.504 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:56.512 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2026-02-17T05:02:56.327843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001704"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084349,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:56.513 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:56.690 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:56.690 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:56.699 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2026-02-17T05:02:56.327843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001704"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084349,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:02:56.700 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:02:56.700 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80027, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:31.816012+0000', 'last_change': '2026-02-17T05:02:31.816012+0000', 'last_active': '2026-02-17T05:02:31.816012+0000', 'last_peered': '2026-02-17T05:02:31.816012+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:31.816012+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:02:56.701 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:02:56.916 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:56.916 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":543,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:02:56.925 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:02:56.926 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:02:57.102 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:02:57.102 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:02:57.111 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2026-02-17T05:02:56.327843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001704"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084349,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:00.113 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:00.287 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:00.288 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:00.296 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2026-02-17T05:02:58.328027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001630"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084349,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:00.297 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:00.478 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:00.478 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:00.486 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2026-02-17T05:03:00.328304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001625"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084350,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:00.487 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:00.488 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80027, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:02:31.816012+0000', 'last_change': '2026-02-17T05:02:31.816012+0000', 'last_active': '2026-02-17T05:02:31.816012+0000', 'last_peered': '2026-02-17T05:02:31.816012+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:02:31.816012+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:00.488 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:00.709 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:00.709 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":547,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:00.718 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:00.718 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:00.896 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:00.896 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:00.904 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2026-02-17T05:03:00.328304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001625"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084350,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:03.905 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:04.081 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:04.081 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:04.090 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2026-02-17T05:03:02.328568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001621"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80027,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:02:31.816012+0000","last_change":"2026-02-17T05:02:31.816012+0000","last_active":"2026-02-17T05:02:31.816012+0000","last_peered":"2026-02-17T05:02:31.816012+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:02:31.816012+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084350,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:04.091 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:05.162 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:05.163 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:05.172 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2026-02-17T05:03:04.328812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001560"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084351,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:05.173 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:05.173 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80031, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:01.816669+0000', 'last_change': '2026-02-17T05:03:01.816669+0000', 'last_active': '2026-02-17T05:03:01.816669+0000', 'last_peered': '2026-02-17T05:03:01.816669+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:01.816669+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:05.173 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:05.390 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:05.391 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":552,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:05.400 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:05.401 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:05.577 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:05.578 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:05.585 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2026-02-17T05:03:04.328812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001560"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084351,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705779,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:08.587 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:08.766 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:08.767 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:08.775 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2026-02-17T05:03:08.329261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001418"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084351,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:08.776 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:08.954 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:08.954 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:08.963 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2026-02-17T05:03:08.329261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001418"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084351,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:08.964 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:08.964 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80031, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:01.816669+0000', 'last_change': '2026-02-17T05:03:01.816669+0000', 'last_active': '2026-02-17T05:03:01.816669+0000', 'last_peered': '2026-02-17T05:03:01.816669+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:01.816669+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:08.965 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:09.184 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:09.184 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":556,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:09.194 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:09.194 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:09.371 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:09.372 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:09.380 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2026-02-17T05:03:08.329261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001418"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084351,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:12.381 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:12.557 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:12.558 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:12.566 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2026-02-17T05:03:12.329712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001408"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084352,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:12.567 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:12.835 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:03:12.834+0000 7f10ca482640 0 --2- 10.20.193.100:0/2438761152 >> [v2:10.20.193.100:3302/0,v1:10.20.193.100:6791/0] conn(0x7f10c4175690 0x7f10c410ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=6 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:03:12.943 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:12.944 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:12.952 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2026-02-17T05:03:12.329712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001408"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084352,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:12.953 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:12.954 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80031, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:01.816669+0000', 'last_change': '2026-02-17T05:03:01.816669+0000', 'last_active': '2026-02-17T05:03:01.816669+0000', 'last_peered': '2026-02-17T05:03:01.816669+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:01.816669+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:12.954 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:13.177 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:13.177 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":560,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:13.186 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:13.187 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:13.362 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:13.362 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:13.371 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2026-02-17T05:03:12.329712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001408"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084352,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705780,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:16.372 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:16.550 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:16.551 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:16.559 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2026-02-17T05:03:16.330154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001341"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:16.561 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:16.735 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:16.735 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:16.744 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2026-02-17T05:03:16.330154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001341"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:16.745 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:16.745 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80031, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:01.816669+0000', 'last_change': '2026-02-17T05:03:01.816669+0000', 'last_active': '2026-02-17T05:03:01.816669+0000', 'last_peered': '2026-02-17T05:03:01.816669+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:01.816669+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:16.745 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:16.966 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:16.967 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":563,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:16.976 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:16.976 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:17.155 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:17.156 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:17.165 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2026-02-17T05:03:16.330154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001341"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:20.167 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:20.344 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:20.345 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:20.353 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2026-02-17T05:03:20.330624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001362"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:20.355 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:20.534 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:20.534 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:20.543 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2026-02-17T05:03:20.330624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001362"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:20.544 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:20.544 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80031, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:01.816669+0000', 'last_change': '2026-02-17T05:03:01.816669+0000', 'last_active': '2026-02-17T05:03:01.816669+0000', 'last_peered': '2026-02-17T05:03:01.816669+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:01.816669+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:20.544 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:20.763 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:20.763 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":567,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:20.772 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:20.772 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:20.954 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:20.954 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:20.962 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2026-02-17T05:03:20.330624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001362"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:23.963 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:24.236 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:24.236 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:24.245 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2026-02-17T05:03:22.330846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001355"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705782,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:24.245 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:24.422 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:24.422 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:24.431 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2026-02-17T05:03:24.331069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001357"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084355,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:24.431 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:24.431 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80031, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:01.816669+0000', 'last_change': '2026-02-17T05:03:01.816669+0000', 'last_active': '2026-02-17T05:03:01.816669+0000', 'last_peered': '2026-02-17T05:03:01.816669+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:01.816669+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:24.431 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:24.648 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:24.649 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":571,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:24.658 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:24.658 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:24.834 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:24.834 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:24.842 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2026-02-17T05:03:24.331069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001357"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084355,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:27.844 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:28.023 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:28.023 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:28.033 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2026-02-17T05:03:26.331341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001405"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084355,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:28.034 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:28.208 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:28.209 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:28.217 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2026-02-17T05:03:26.331341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001405"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084355,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:28.218 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:28.218 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80031, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:01.816669+0000', 'last_change': '2026-02-17T05:03:01.816669+0000', 'last_active': '2026-02-17T05:03:01.816669+0000', 'last_peered': '2026-02-17T05:03:01.816669+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:01.816669+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:28.219 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:28.438 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:28.438 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":575,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:28.448 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:28.448 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:28.623 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:28.623 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:28.631 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2026-02-17T05:03:28.331631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001476"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084355,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:31.633 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:31.810 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:31.811 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:31.819 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2026-02-17T05:03:30.331899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:31.820 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:31.993 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:31.993 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:32.002 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2026-02-17T05:03:30.331899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:32.003 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:32.003 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80031, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:01.816669+0000', 'last_change': '2026-02-17T05:03:01.816669+0000', 'last_active': '2026-02-17T05:03:01.816669+0000', 'last_peered': '2026-02-17T05:03:01.816669+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:01.816669+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:32.003 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:32.222 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:32.222 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":579,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:32.232 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:32.233 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:32.408 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:32.408 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:32.417 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2026-02-17T05:03:32.332216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001592"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80031,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:01.816669+0000","last_change":"2026-02-17T05:03:01.816669+0000","last_active":"2026-02-17T05:03:01.816669+0000","last_peered":"2026-02-17T05:03:01.816669+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:01.816669+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705784,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:35.419 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:35.596 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:35.597 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:35.605 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2026-02-17T05:03:34.332463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084357,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:35.606 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:35.785 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:35.786 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:35.794 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2026-02-17T05:03:34.332463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084357,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:35.795 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:35.795 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80035, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:31.817209+0000', 'last_change': '2026-02-17T05:03:31.817209+0000', 'last_active': '2026-02-17T05:03:31.817209+0000', 'last_peered': '2026-02-17T05:03:31.817209+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:31.817209+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:35.795 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:36.011 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:36.012 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":582,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:36.021 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:36.021 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:36.200 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:36.200 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:36.208 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2026-02-17T05:03:34.332463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084357,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:39.210 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:39.386 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:39.386 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:39.396 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2026-02-17T05:03:38.333006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001665"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084357,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:39.398 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:39.573 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:39.573 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:39.581 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2026-02-17T05:03:38.333006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001665"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084357,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:39.581 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:39.581 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80035, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:31.817209+0000', 'last_change': '2026-02-17T05:03:31.817209+0000', 'last_active': '2026-02-17T05:03:31.817209+0000', 'last_peered': '2026-02-17T05:03:31.817209+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:31.817209+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:39.582 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:39.797 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:39.797 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":586,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:39.805 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:39.805 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:39.982 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:39.982 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:39.990 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2026-02-17T05:03:38.333006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001665"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084357,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:42.991 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:43.167 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:43.167 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:43.175 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2026-02-17T05:03:42.333515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001616"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:43.175 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:43.349 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:43.349 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:43.357 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2026-02-17T05:03:42.333515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001616"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:43.357 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:43.357 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80035, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:31.817209+0000', 'last_change': '2026-02-17T05:03:31.817209+0000', 'last_active': '2026-02-17T05:03:31.817209+0000', 'last_peered': '2026-02-17T05:03:31.817209+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:31.817209+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:43.357 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:43.574 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:43.574 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":590,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:43.584 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:43.585 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:43.760 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:43.760 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:43.769 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2026-02-17T05:03:42.333515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001616"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:46.772 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:46.950 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:46.950 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:46.959 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2026-02-17T05:03:46.334115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001651"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:46.960 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:47.133 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:47.133 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:47.142 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2026-02-17T05:03:46.334115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001651"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:47.143 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:47.143 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80035, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:31.817209+0000', 'last_change': '2026-02-17T05:03:31.817209+0000', 'last_active': '2026-02-17T05:03:31.817209+0000', 'last_peered': '2026-02-17T05:03:31.817209+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:31.817209+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:47.144 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:47.362 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:47.362 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":594,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:47.372 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:47.372 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:47.547 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:47.548 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:47.556 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2026-02-17T05:03:46.334115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001651"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:50.557 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:50.734 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:50.735 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:50.743 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2026-02-17T05:03:50.334708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:50.745 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:50.921 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:50.921 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:50.930 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2026-02-17T05:03:50.334708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:50.931 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:50.931 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80035, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:31.817209+0000', 'last_change': '2026-02-17T05:03:31.817209+0000', 'last_active': '2026-02-17T05:03:31.817209+0000', 'last_peered': '2026-02-17T05:03:31.817209+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:31.817209+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:50.931 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:51.148 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:51.149 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":598,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:51.158 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:51.158 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:51.335 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:51.335 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:51.344 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2026-02-17T05:03:50.334708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:54.346 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:54.526 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:54.527 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:54.536 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2026-02-17T05:03:54.335334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001818"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:54.538 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:55.608 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:55.609 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:55.618 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2026-02-17T05:03:54.335334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001818"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:55.619 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:55.619 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80035, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:31.817209+0000', 'last_change': '2026-02-17T05:03:31.817209+0000', 'last_active': '2026-02-17T05:03:31.817209+0000', 'last_peered': '2026-02-17T05:03:31.817209+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:31.817209+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:55.620 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:55.840 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:55.840 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":602,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:55.850 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:55.850 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:56.028 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:56.028 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:56.037 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2026-02-17T05:03:54.335334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001818"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:59.038 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:59.216 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:59.216 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:59.225 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2026-02-17T05:03:58.335872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001757"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:59.226 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:59.404 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:59.405 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:59.413 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2026-02-17T05:03:58.335872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001757"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:03:59.414 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:03:59.414 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80035, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:31.817209+0000', 'last_change': '2026-02-17T05:03:31.817209+0000', 'last_active': '2026-02-17T05:03:31.817209+0000', 'last_peered': '2026-02-17T05:03:31.817209+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:31.817209+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:03:59.414 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:03:59.632 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:59.633 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":606,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:03:59.642 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:03:59.642 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:03:59.820 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:03:59.820 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:03:59.829 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2026-02-17T05:03:58.335872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001757"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T04:59:58.454936+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T04:59:58.454936+0000","last_peered":"2026-02-17T04:59:58.454936+0000","last_clean":"2026-02-17T04:59:58.454936+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T04:59:58.454936+0000","last_undegraded":"2026-02-17T04:59:58.454936+0000","last_fullsized":"2026-02-17T04:59:58.454936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:02.830 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:03.006 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:03.006 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:03.015 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2026-02-17T05:04:02.336323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001614"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:03.015 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:03.189 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:03.190 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:03.199 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2026-02-17T05:04:02.336323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001614"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:03.200 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:03.200 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80035, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:03:31.817209+0000', 'last_change': '2026-02-17T05:03:31.817209+0000', 'last_active': '2026-02-17T05:03:31.817209+0000', 'last_peered': '2026-02-17T05:03:31.817209+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:03:31.817209+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:03.200 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:03.421 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:03.422 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":610,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:03.432 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:03.432 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:03.610 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:03.610 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:03.618 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2026-02-17T05:04:02.336323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001614"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80035,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:03:31.817209+0000","last_change":"2026-02-17T05:03:31.817209+0000","last_active":"2026-02-17T05:03:31.817209+0000","last_peered":"2026-02-17T05:03:31.817209+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:03:31.817209+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:06.619 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:06.797 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:06.797 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:06.807 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2026-02-17T05:04:06.336768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001434"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80039,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817679+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817679+0000","last_peered":"2026-02-17T05:04:01.817679+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817679+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084363,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:06.808 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:06.986 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:06.987 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:06.995 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2026-02-17T05:04:06.336768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001434"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80039,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817679+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817679+0000","last_peered":"2026-02-17T05:04:01.817679+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817679+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084363,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:06.996 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:06.997 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80039, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:01.817679+0000', 'last_change': '2026-02-17T05:04:01.817679+0000', 'last_active': '2026-02-17T05:04:01.817679+0000', 'last_peered': '2026-02-17T05:04:01.817679+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:01.817679+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:06.997 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:07.215 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:07.215 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":614,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:07.225 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:07.225 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:07.399 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:07.400 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:07.407 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2026-02-17T05:04:06.336768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001434"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80039,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817679+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817679+0000","last_peered":"2026-02-17T05:04:01.817679+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817679+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084363,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:10.410 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:10.586 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:10.587 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:10.595 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2026-02-17T05:04:10.337230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001357"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80039,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817679+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817679+0000","last_peered":"2026-02-17T05:04:01.817679+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817679+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:10.596 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:10.774 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:10.774 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:10.783 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2026-02-17T05:04:10.337230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001357"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80039,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817679+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817679+0000","last_peered":"2026-02-17T05:04:01.817679+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817679+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:10.784 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:10.784 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80039, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:01.817679+0000', 'last_change': '2026-02-17T05:04:01.817679+0000', 'last_active': '2026-02-17T05:04:01.817679+0000', 'last_peered': '2026-02-17T05:04:01.817679+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:01.817679+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:10.784 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:11.001 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:11.001 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":617,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:11.011 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:11.011 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:11.186 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:11.187 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:11.196 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2026-02-17T05:04:10.337230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001357"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80039,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817679+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817679+0000","last_peered":"2026-02-17T05:04:01.817679+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817679+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:14.197 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:14.381 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:14.381 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:14.390 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2026-02-17T05:04:14.337826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001503"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084365,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:14.391 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:14.565 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:14.565 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:14.573 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2026-02-17T05:04:14.337826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001503"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084365,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:14.575 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:14.575 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80040, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:01.817711+0000', 'last_change': '2026-02-17T05:04:01.817679+0000', 'last_active': '2026-02-17T05:04:01.817711+0000', 'last_peered': '2026-02-17T05:04:01.817711+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:01.817711+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:14.575 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:14.799 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:14.799 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":621,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:14.808 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:14.808 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:14.982 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:14.982 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:14.990 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2026-02-17T05:04:14.337826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001503"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084365,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:17.992 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:18.261 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:04:18.260+0000 7f9b79952640 0 --2- 10.20.193.100:0/1345947571 >> v2:10.20.193.100:3300/0 conn(0x7f9b741373c0 0x7f9b74137790 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=10187693987663364158 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:04:18.369 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:18.370 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:18.378 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2026-02-17T05:04:18.338316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001547"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084365,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:18.379 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:18.556 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:18.557 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:18.565 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2026-02-17T05:04:18.338316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001547"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084365,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:18.566 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:18.566 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80040, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:01.817711+0000', 'last_change': '2026-02-17T05:04:01.817679+0000', 'last_active': '2026-02-17T05:04:01.817711+0000', 'last_peered': '2026-02-17T05:04:01.817711+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:01.817711+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:18.567 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:18.780 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:18.780 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":625,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:18.789 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:18.789 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:18.974 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:18.974 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:18.983 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2026-02-17T05:04:18.338316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001547"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084365,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:21.984 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:22.159 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:22.159 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:22.167 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2026-02-17T05:04:20.338610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001623"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:22.168 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:22.346 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:22.347 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:22.356 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2026-02-17T05:04:22.338958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001727"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:22.357 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:22.357 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80040, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:01.817711+0000', 'last_change': '2026-02-17T05:04:01.817679+0000', 'last_active': '2026-02-17T05:04:01.817711+0000', 'last_peered': '2026-02-17T05:04:01.817711+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:01.817711+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:22.357 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:22.575 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:22.575 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":629,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:22.585 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:22.585 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:22.758 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:22.758 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:22.766 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2026-02-17T05:04:22.338958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001727"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:25.767 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:25.948 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:25.948 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:25.956 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2026-02-17T05:04:24.339200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001674"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:25.956 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:26.130 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:26.130 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:26.138 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2026-02-17T05:04:24.339200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001674"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:26.139 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:26.139 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80040, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:01.817711+0000', 'last_change': '2026-02-17T05:04:01.817679+0000', 'last_active': '2026-02-17T05:04:01.817711+0000', 'last_peered': '2026-02-17T05:04:01.817711+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:01.817711+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:26.140 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:26.357 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:26.357 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":633,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:26.367 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:26.367 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:26.543 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:26.544 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:26.553 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2026-02-17T05:04:26.339471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001644"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:29.554 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:29.728 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:29.729 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:29.737 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2026-02-17T05:04:28.339714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:29.738 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:29.913 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:29.914 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:29.922 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2026-02-17T05:04:28.339714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:29.923 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:29.923 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80040, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:01.817711+0000', 'last_change': '2026-02-17T05:04:01.817679+0000', 'last_active': '2026-02-17T05:04:01.817711+0000', 'last_peered': '2026-02-17T05:04:01.817711+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:01.817711+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:29.924 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:30.140 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:30.141 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":637,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:30.150 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:30.151 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:30.327 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:30.327 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:30.335 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2026-02-17T05:04:28.339714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:33.337 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:33.513 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:33.513 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:33.522 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2026-02-17T05:04:32.340209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084368,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:33.523 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:33.703 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:33.703 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:33.712 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2026-02-17T05:04:32.340209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084368,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:33.713 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:33.713 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80040, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:01.817711+0000', 'last_change': '2026-02-17T05:04:01.817679+0000', 'last_active': '2026-02-17T05:04:01.817711+0000', 'last_peered': '2026-02-17T05:04:01.817711+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:01.817711+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:33.713 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:33.928 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:33.928 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":640,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:33.938 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:33.938 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:34.114 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:34.114 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:34.122 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2026-02-17T05:04:32.340209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80040,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:01.817711+0000","last_change":"2026-02-17T05:04:01.817679+0000","last_active":"2026-02-17T05:04:01.817711+0000","last_peered":"2026-02-17T05:04:01.817711+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:01.817711+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084368,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:37.123 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:37.305 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:37.306 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:37.314 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2026-02-17T05:04:36.340626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001426"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084369,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:37.315 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:37.493 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:37.493 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:37.502 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2026-02-17T05:04:36.340626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001426"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084369,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:37.503 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:37.504 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80044, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:31.818192+0000', 'last_change': '2026-02-17T05:04:31.818192+0000', 'last_active': '2026-02-17T05:04:31.818192+0000', 'last_peered': '2026-02-17T05:04:31.818192+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:31.818192+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:37.504 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:37.724 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:37.725 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":644,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:37.733 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:37.733 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:37.909 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:37.909 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:37.918 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2026-02-17T05:04:36.340626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001426"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084369,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:40.921 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:41.098 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:41.098 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:41.106 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2026-02-17T05:04:40.341037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001322"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:41.107 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:41.286 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:41.286 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:41.295 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2026-02-17T05:04:40.341037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001322"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:41.296 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:41.296 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80044, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:31.818192+0000', 'last_change': '2026-02-17T05:04:31.818192+0000', 'last_active': '2026-02-17T05:04:31.818192+0000', 'last_peered': '2026-02-17T05:04:31.818192+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:31.818192+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:41.296 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:41.515 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:41.516 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":648,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:41.526 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:41.526 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:41.708 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:41.709 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:41.717 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2026-02-17T05:04:40.341037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001322"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705798,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:44.719 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:44.902 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:44.903 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:44.911 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2026-02-17T05:04:44.341486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001276"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:44.912 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:45.182 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:45.183 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:45.191 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2026-02-17T05:04:44.341486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001276"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:45.192 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:45.192 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80044, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:31.818192+0000', 'last_change': '2026-02-17T05:04:31.818192+0000', 'last_active': '2026-02-17T05:04:31.818192+0000', 'last_peered': '2026-02-17T05:04:31.818192+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:31.818192+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:45.192 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:45.410 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:45.411 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":652,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:45.420 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:45.420 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:45.597 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:45.597 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:45.606 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2026-02-17T05:04:44.341486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001276"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:48.607 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:48.780 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:48.781 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:48.789 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2026-02-17T05:04:48.342026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001399"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:48.790 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:48.965 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:48.965 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:48.974 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2026-02-17T05:04:48.342026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001399"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:48.975 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:48.975 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80044, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:31.818192+0000', 'last_change': '2026-02-17T05:04:31.818192+0000', 'last_active': '2026-02-17T05:04:31.818192+0000', 'last_peered': '2026-02-17T05:04:31.818192+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:31.818192+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:48.976 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:49.196 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:49.196 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":656,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:49.207 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:49.207 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:49.379 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:49.380 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:49.387 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2026-02-17T05:04:48.342026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001399"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:52.390 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:52.567 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:52.568 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:52.576 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2026-02-17T05:04:52.342635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:52.577 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:52.751 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:52.751 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:52.759 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2026-02-17T05:04:52.342635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:52.760 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:52.760 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80044, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:31.818192+0000', 'last_change': '2026-02-17T05:04:31.818192+0000', 'last_active': '2026-02-17T05:04:31.818192+0000', 'last_peered': '2026-02-17T05:04:31.818192+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:31.818192+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:52.761 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:52.979 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:52.979 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":659,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:52.989 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:52.989 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:53.165 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:53.165 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:53.173 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2026-02-17T05:04:52.342635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:56.175 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:56.354 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:56.355 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:56.365 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2026-02-17T05:04:56.343207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001721"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:56.366 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:56.542 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:56.543 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:56.551 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2026-02-17T05:04:56.343207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001721"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:56.553 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:04:56.553 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80044, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:31.818192+0000', 'last_change': '2026-02-17T05:04:31.818192+0000', 'last_active': '2026-02-17T05:04:31.818192+0000', 'last_peered': '2026-02-17T05:04:31.818192+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:31.818192+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:04:56.553 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:04:56.769 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:56.769 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":663,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:04:56.778 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:04:56.778 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:04:56.955 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:04:56.956 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:04:56.965 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2026-02-17T05:04:56.343207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001721"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:04:59.966 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:00.148 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:00.148 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:00.156 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2026-02-17T05:04:58.343459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001722"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:00.158 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:00.335 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:00.335 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:00.343 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2026-02-17T05:04:58.343459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001722"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:00.344 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:00.344 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80044, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:31.818192+0000', 'last_change': '2026-02-17T05:04:31.818192+0000', 'last_active': '2026-02-17T05:04:31.818192+0000', 'last_peered': '2026-02-17T05:04:31.818192+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:31.818192+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:00.344 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:00.611 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:05:00.610+0000 7f31b2ffd640 0 --2- 10.20.193.100:0/2115119448 >> v2:10.20.193.100:3300/0 conn(0x7f31b4084770 0x7f31b4178390 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:05:00.761 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:00.761 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":667,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:00.770 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:00.770 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:00.947 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:00.947 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:00.956 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2026-02-17T05:05:00.343752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001725"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:03.957 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:04.131 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:04.131 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:04.140 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2026-02-17T05:05:02.343987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001667"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:04.141 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:04.316 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:04.317 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:04.326 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2026-02-17T05:05:02.343987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001667"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80044,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:04:31.818192+0000","last_change":"2026-02-17T05:04:31.818192+0000","last_active":"2026-02-17T05:04:31.818192+0000","last_peered":"2026-02-17T05:04:31.818192+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:04:31.818192+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705802,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:04.327 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:04.327 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80044, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:04:31.818192+0000', 'last_change': '2026-02-17T05:04:31.818192+0000', 'last_active': '2026-02-17T05:04:31.818192+0000', 'last_peered': '2026-02-17T05:04:31.818192+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:04:31.818192+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:04.327 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:04.548 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:04.548 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":671,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:04.558 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:04.558 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:04.735 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:04.736 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:04.744 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2026-02-17T05:05:04.344251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:07.746 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:07.926 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:07.926 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:07.934 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2026-02-17T05:05:06.344526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:07.935 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:08.111 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:08.111 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:08.120 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2026-02-17T05:05:06.344526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:08.121 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:08.121 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80048, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:01.818883+0000', 'last_change': '2026-02-17T05:05:01.818883+0000', 'last_active': '2026-02-17T05:05:01.818883+0000', 'last_peered': '2026-02-17T05:05:01.818883+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:01.818883+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:08.122 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:08.339 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:08.339 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":675,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:08.348 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:08.348 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:08.524 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:08.525 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:08.534 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2026-02-17T05:05:08.344825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:11.536 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:11.713 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:11.714 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:11.722 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2026-02-17T05:05:10.345058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084376,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:11.722 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:11.900 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:11.901 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:11.909 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2026-02-17T05:05:10.345058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084376,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:11.909 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:11.910 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80048, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:01.818883+0000', 'last_change': '2026-02-17T05:05:01.818883+0000', 'last_active': '2026-02-17T05:05:01.818883+0000', 'last_peered': '2026-02-17T05:05:01.818883+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:01.818883+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:11.910 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:12.126 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:12.126 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":679,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":10,"modified":"2026-02-17T05:02:24.323664+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:12.135 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:12.135 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:12.310 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:12.310 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:12.319 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2026-02-17T05:05:10.345058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084376,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:15.320 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:15.502 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:15.502 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:15.513 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2026-02-17T05:05:14.345516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001529"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:15.514 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:15.690 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:15.691 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:15.700 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2026-02-17T05:05:14.345516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001529"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:15.701 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:15.701 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80048, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:01.818883+0000', 'last_change': '2026-02-17T05:05:01.818883+0000', 'last_active': '2026-02-17T05:05:01.818883+0000', 'last_peered': '2026-02-17T05:05:01.818883+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:01.818883+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:15.702 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:15.925 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:15.925 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":682,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-17T05:05:14.345560+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:05:15.935 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:15.935 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:16.209 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:16.209 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:16.217 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2026-02-17T05:05:14.345516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001529"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:19.220 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:19.394 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:19.394 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:19.402 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2026-02-17T05:05:18.346002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001475"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:19.402 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:19.578 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:19.579 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:19.588 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2026-02-17T05:05:18.346002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001475"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:19.589 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:19.589 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80048, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:01.818883+0000', 'last_change': '2026-02-17T05:05:01.818883+0000', 'last_active': '2026-02-17T05:05:01.818883+0000', 'last_peered': '2026-02-17T05:05:01.818883+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:01.818883+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:19.589 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:19.808 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:19.808 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":686,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-17T05:05:14.345560+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:05:19.817 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:19.818 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:19.995 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:19.995 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:20.004 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2026-02-17T05:05:18.346002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001475"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:23.005 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:23.182 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:23.183 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:23.191 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2026-02-17T05:05:22.346469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001411"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084378,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:23.192 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:23.369 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:23.369 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:23.378 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2026-02-17T05:05:22.346469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001411"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084378,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:23.379 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:23.379 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80048, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:01.818883+0000', 'last_change': '2026-02-17T05:05:01.818883+0000', 'last_active': '2026-02-17T05:05:01.818883+0000', 'last_peered': '2026-02-17T05:05:01.818883+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:01.818883+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:23.380 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:23.601 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:23.601 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":690,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":11,"modified":"2026-02-17T05:05:14.345560+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:05:23.610 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:23.611 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:23.787 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:23.788 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:23.796 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2026-02-17T05:05:22.346469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001411"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084378,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705806,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:26.798 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:26.977 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:26.977 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:26.986 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2026-02-17T05:05:26.347081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001565"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:26.987 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:27.164 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:27.165 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:27.173 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2026-02-17T05:05:26.347081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001565"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:27.174 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:27.174 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80048, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:01.818883+0000', 'last_change': '2026-02-17T05:05:01.818883+0000', 'last_active': '2026-02-17T05:05:01.818883+0000', 'last_peered': '2026-02-17T05:05:01.818883+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:01.818883+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:27.174 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:27.443 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:05:27.442+0000 7f5636dd1640 0 --2- 10.20.193.100:0/1496787127 >> [v2:10.20.193.100:3300/0,v1:10.20.193.100:6789/0] conn(0x7f5630175690 0x7f563010c780 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:05:27.592 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:27.592 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":694,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:27.601 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:27.601 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:27.775 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:27.776 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:27.783 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2026-02-17T05:05:26.347081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001565"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705807,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:30.785 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:30.962 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:30.963 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:30.970 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2026-02-17T05:05:30.347672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001670"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:30.972 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:31.147 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:31.148 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:31.156 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2026-02-17T05:05:30.347672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001670"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:31.157 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:31.157 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80048, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:01.818883+0000', 'last_change': '2026-02-17T05:05:01.818883+0000', 'last_active': '2026-02-17T05:05:01.818883+0000', 'last_peered': '2026-02-17T05:05:01.818883+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:01.818883+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:31.158 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:31.377 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:31.378 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":698,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:31.387 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:31.388 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:31.562 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:31.563 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:31.572 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2026-02-17T05:05:30.347672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001670"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80048,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:01.818883+0000","last_change":"2026-02-17T05:05:01.818883+0000","last_active":"2026-02-17T05:05:01.818883+0000","last_peered":"2026-02-17T05:05:01.818883+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:01.818883+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:34.574 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:34.752 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:34.752 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:34.761 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2026-02-17T05:05:34.348312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001842"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:34.762 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:34.943 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:34.944 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:34.952 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2026-02-17T05:05:34.348312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001842"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:34.953 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:34.953 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80052, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:31.819548+0000', 'last_change': '2026-02-17T05:05:31.819548+0000', 'last_active': '2026-02-17T05:05:31.819548+0000', 'last_peered': '2026-02-17T05:05:31.819548+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:31.819548+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:34.953 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:35.174 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:35.174 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":702,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:35.184 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:35.184 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:35.359 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:35.359 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:35.367 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2026-02-17T05:05:34.348312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001842"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705809,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:38.369 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:38.543 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:38.543 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:38.553 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2026-02-17T05:05:38.348877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001795"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:38.554 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:38.733 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:38.733 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:38.742 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2026-02-17T05:05:38.348877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001795"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:38.743 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:38.743 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80052, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:31.819548+0000', 'last_change': '2026-02-17T05:05:31.819548+0000', 'last_active': '2026-02-17T05:05:31.819548+0000', 'last_peered': '2026-02-17T05:05:31.819548+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:31.819548+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:38.744 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:38.963 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:38.963 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":705,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:38.972 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:38.973 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:39.149 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:39.149 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:39.158 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2026-02-17T05:05:38.348877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001795"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:42.159 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:42.336 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:42.336 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:42.344 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2026-02-17T05:05:40.349103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001733"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084382,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:42.345 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:42.522 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:42.522 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:42.531 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2026-02-17T05:05:42.349466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001794"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084382,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:42.532 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:42.532 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80052, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:31.819548+0000', 'last_change': '2026-02-17T05:05:31.819548+0000', 'last_active': '2026-02-17T05:05:31.819548+0000', 'last_peered': '2026-02-17T05:05:31.819548+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:31.819548+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:42.532 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:42.750 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:42.750 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":709,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:42.760 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:42.760 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:42.933 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:42.934 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:42.942 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2026-02-17T05:05:42.349466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001794"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084382,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:45.944 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:46.121 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:46.122 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:46.130 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2026-02-17T05:05:44.349664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001706"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084383,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:46.131 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:46.304 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:46.304 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:46.312 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2026-02-17T05:05:44.349664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001706"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084383,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:46.313 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:46.313 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80052, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:31.819548+0000', 'last_change': '2026-02-17T05:05:31.819548+0000', 'last_active': '2026-02-17T05:05:31.819548+0000', 'last_peered': '2026-02-17T05:05:31.819548+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:31.819548+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:46.313 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:46.539 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:46.539 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":713,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:46.549 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:46.549 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:46.726 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:46.727 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:46.735 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2026-02-17T05:05:46.349921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001609"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084383,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:49.737 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:49.916 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:49.917 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:49.925 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2026-02-17T05:05:48.350145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084383,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:49.927 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:50.104 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:50.104 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:50.112 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2026-02-17T05:05:48.350145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084383,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:50.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:50.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80052, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:31.819548+0000', 'last_change': '2026-02-17T05:05:31.819548+0000', 'last_active': '2026-02-17T05:05:31.819548+0000', 'last_peered': '2026-02-17T05:05:31.819548+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:31.819548+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:50.113 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:50.326 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:50.327 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":717,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:50.336 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:50.336 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:50.510 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:50.510 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:50.519 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2026-02-17T05:05:50.350398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001520"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:53.521 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:53.696 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:53.697 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:53.705 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2026-02-17T05:05:52.350622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001518"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:53.706 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:53.882 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:53.882 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:53.891 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2026-02-17T05:05:52.350622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001518"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:53.892 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:53.892 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80052, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:31.819548+0000', 'last_change': '2026-02-17T05:05:31.819548+0000', 'last_active': '2026-02-17T05:05:31.819548+0000', 'last_peered': '2026-02-17T05:05:31.819548+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:31.819548+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:53.892 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:54.110 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:54.110 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":721,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:54.119 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:54.119 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:54.299 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:54.299 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:54.308 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2026-02-17T05:05:52.350622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001518"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:57.310 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:57.486 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:57.486 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:57.494 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2026-02-17T05:05:56.351047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001382"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084385,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:57.495 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:57.671 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:57.672 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:57.680 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2026-02-17T05:05:56.351047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001382"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084385,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:05:57.681 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:05:57.681 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80052, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:31.819548+0000', 'last_change': '2026-02-17T05:05:31.819548+0000', 'last_active': '2026-02-17T05:05:31.819548+0000', 'last_peered': '2026-02-17T05:05:31.819548+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:31.819548+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:05:57.682 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:05:57.898 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:57.898 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":724,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:05:57.908 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:05:57.908 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:05:58.082 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:05:58.082 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:05:58.090 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2026-02-17T05:05:56.351047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001382"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:01:58.464338+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:01:58.464338+0000","last_peered":"2026-02-17T05:01:58.464338+0000","last_clean":"2026-02-17T05:01:58.464338+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:01:58.464338+0000","last_undegraded":"2026-02-17T05:01:58.464338+0000","last_fullsized":"2026-02-17T05:01:58.464338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084385,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:01.092 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:01.275 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:01.275 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:01.283 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2026-02-17T05:06:00.351432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001286"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084386,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:01.284 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:01.464 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:01.464 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:01.473 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2026-02-17T05:06:00.351432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001286"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084386,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:01.474 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:01.474 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80052, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:05:31.819548+0000', 'last_change': '2026-02-17T05:05:31.819548+0000', 'last_active': '2026-02-17T05:05:31.819548+0000', 'last_peered': '2026-02-17T05:05:31.819548+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:05:31.819548+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:01.475 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:01.691 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:01.691 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":728,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:01.700 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:01.701 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:01.876 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:01.877 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:01.885 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2026-02-17T05:06:00.351432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001286"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80052,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:05:31.819548+0000","last_change":"2026-02-17T05:05:31.819548+0000","last_active":"2026-02-17T05:05:31.819548+0000","last_peered":"2026-02-17T05:05:31.819548+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:05:31.819548+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084386,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:04.887 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:05.065 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:05.066 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:05.074 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2026-02-17T05:06:04.351851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001229"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80056,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819937+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819937+0000","last_peered":"2026-02-17T05:06:01.819937+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819937+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:05.075 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:05.248 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:05.249 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:05.258 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2026-02-17T05:06:04.351851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001229"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80056,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819937+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819937+0000","last_peered":"2026-02-17T05:06:01.819937+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819937+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:05.259 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:05.259 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80056, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:01.819937+0000', 'last_change': '2026-02-17T05:06:01.819937+0000', 'last_active': '2026-02-17T05:06:01.819937+0000', 'last_peered': '2026-02-17T05:06:01.819937+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:01.819937+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:05.260 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:05.481 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:05.481 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":732,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:05.490 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:05.491 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:05.669 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:05.670 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:05.679 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2026-02-17T05:06:04.351851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001229"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80056,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819937+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819937+0000","last_peered":"2026-02-17T05:06:01.819937+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819937+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:08.681 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:08.857 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:08.858 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:08.867 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2026-02-17T05:06:08.352286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001239"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80056,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819937+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819937+0000","last_peered":"2026-02-17T05:06:01.819937+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819937+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:08.868 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:09.046 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:09.047 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:09.055 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2026-02-17T05:06:08.352286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001239"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80056,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819937+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819937+0000","last_peered":"2026-02-17T05:06:01.819937+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819937+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:09.055 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:09.055 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80056, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:01.819937+0000', 'last_change': '2026-02-17T05:06:01.819937+0000', 'last_active': '2026-02-17T05:06:01.819937+0000', 'last_peered': '2026-02-17T05:06:01.819937+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:01.819937+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:09.056 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:09.271 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:09.271 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":736,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:09.280 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:09.281 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:09.456 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:09.456 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:09.465 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2026-02-17T05:06:08.352286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001239"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80056,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819937+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819937+0000","last_peered":"2026-02-17T05:06:01.819937+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819937+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:12.466 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:12.641 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:12.641 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:12.650 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2026-02-17T05:06:12.352787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001355"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80056,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819937+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819937+0000","last_peered":"2026-02-17T05:06:01.819937+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819937+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084388,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:12.651 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:12.923 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:12.923 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:12.932 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2026-02-17T05:06:12.352787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001355"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80056,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819937+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819937+0000","last_peered":"2026-02-17T05:06:01.819937+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819937+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084388,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:12.933 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:12.933 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80056, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:01.819937+0000', 'last_change': '2026-02-17T05:06:01.819937+0000', 'last_active': '2026-02-17T05:06:01.819937+0000', 'last_peered': '2026-02-17T05:06:01.819937+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:01.819937+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:12.933 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:13.149 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:13.150 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":740,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:13.160 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:13.160 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:13.337 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:13.337 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:13.346 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2026-02-17T05:06:12.352787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001355"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80056,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819937+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819937+0000","last_peered":"2026-02-17T05:06:01.819937+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819937+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084388,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:16.348 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:16.525 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:16.526 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:16.534 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2026-02-17T05:06:16.353295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001443"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084389,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:16.535 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:16.710 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:16.711 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:16.720 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2026-02-17T05:06:16.353295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001443"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084389,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:16.721 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:16.721 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80057, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:01.819967+0000', 'last_change': '2026-02-17T05:06:01.819937+0000', 'last_active': '2026-02-17T05:06:01.819967+0000', 'last_peered': '2026-02-17T05:06:01.819967+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:01.819967+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:16.721 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:16.938 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:16.938 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":743,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:16.947 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:16.948 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:17.127 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:17.128 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:17.138 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2026-02-17T05:06:16.353295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001443"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084389,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:20.140 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:20.319 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:20.319 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:20.327 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2026-02-17T05:06:18.353486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001450"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084389,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:20.328 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:20.504 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:20.504 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:20.512 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2026-02-17T05:06:20.353687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001400"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084390,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:20.513 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:20.513 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80057, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:01.819967+0000', 'last_change': '2026-02-17T05:06:01.819937+0000', 'last_active': '2026-02-17T05:06:01.819967+0000', 'last_peered': '2026-02-17T05:06:01.819967+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:01.819967+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:20.513 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:20.730 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:20.731 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":747,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:20.739 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:20.740 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:20.917 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:20.917 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:20.926 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2026-02-17T05:06:20.353687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001400"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084390,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:23.927 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:24.106 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:24.106 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:24.114 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2026-02-17T05:06:22.353893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001400"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084390,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:24.115 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:24.292 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:24.293 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:24.301 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2026-02-17T05:06:22.353893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001400"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084390,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:24.302 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:24.302 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80057, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:01.819967+0000', 'last_change': '2026-02-17T05:06:01.819937+0000', 'last_active': '2026-02-17T05:06:01.819967+0000', 'last_peered': '2026-02-17T05:06:01.819967+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:01.819967+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:24.303 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:24.523 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:24.524 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":751,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:24.533 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:24.533 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:24.708 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:24.708 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:24.716 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2026-02-17T05:06:24.354123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001336"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084391,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:27.718 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:27.897 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:27.898 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:27.907 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2026-02-17T05:06:26.354332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001345"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084391,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:27.908 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:28.085 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:28.085 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:28.094 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2026-02-17T05:06:26.354332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001345"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084391,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705819,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:28.095 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:28.096 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80057, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:01.819967+0000', 'last_change': '2026-02-17T05:06:01.819937+0000', 'last_active': '2026-02-17T05:06:01.819967+0000', 'last_peered': '2026-02-17T05:06:01.819967+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:01.819967+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:28.096 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:28.311 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:28.311 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":755,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:28.320 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:28.320 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:28.498 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:28.499 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:28.508 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2026-02-17T05:06:28.354534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001239"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084391,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:31.510 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:31.691 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:31.691 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:31.699 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2026-02-17T05:06:30.354761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001275"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084392,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:31.700 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:31.967 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:06:31.966+0000 7fdd3da42640 0 --2- 10.20.193.100:0/1282568364 >> v2:10.20.193.100:3300/0 conn(0x7fdd38178600 0x7fdd381989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=8729426535675340056 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:06:32.078 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:32.079 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:32.087 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2026-02-17T05:06:30.354761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001275"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084392,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:32.088 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:32.089 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80057, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:01.819967+0000', 'last_change': '2026-02-17T05:06:01.819937+0000', 'last_active': '2026-02-17T05:06:01.819967+0000', 'last_peered': '2026-02-17T05:06:01.819967+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:01.819967+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:32.089 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:32.314 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:32.314 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":759,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:32.324 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:32.324 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:32.501 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:32.501 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:32.511 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2026-02-17T05:06:32.354992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001305"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80057,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:01.819967+0000","last_change":"2026-02-17T05:06:01.819937+0000","last_active":"2026-02-17T05:06:01.819967+0000","last_peered":"2026-02-17T05:06:01.819967+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:01.819967+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084392,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:35.512 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:35.687 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:35.687 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:35.695 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2026-02-17T05:06:34.355207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001313"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084393,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:35.697 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:35.872 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:35.873 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:35.881 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2026-02-17T05:06:34.355207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001313"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084393,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:35.882 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:35.882 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80061, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:31.820443+0000', 'last_change': '2026-02-17T05:06:31.820443+0000', 'last_active': '2026-02-17T05:06:31.820443+0000', 'last_peered': '2026-02-17T05:06:31.820443+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:31.820443+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:35.883 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:36.105 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:36.105 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":763,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:36.114 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:36.114 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:36.289 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:36.289 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:36.297 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2026-02-17T05:06:34.355207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001313"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084393,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:39.298 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:39.474 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:39.475 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:39.484 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2026-02-17T05:06:38.355976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001643"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084393,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:39.485 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:39.663 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:39.663 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:39.671 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2026-02-17T05:06:38.355976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001643"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084393,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:39.671 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:39.671 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80061, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:31.820443+0000', 'last_change': '2026-02-17T05:06:31.820443+0000', 'last_active': '2026-02-17T05:06:31.820443+0000', 'last_peered': '2026-02-17T05:06:31.820443+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:31.820443+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:39.672 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:39.890 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:39.890 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":766,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:39.899 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:39.900 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:40.074 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:40.074 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:40.082 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2026-02-17T05:06:38.355976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001643"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084393,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:43.084 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:43.262 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:43.263 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:43.271 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2026-02-17T05:06:42.356497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001735"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084394,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:43.272 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:43.450 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:43.450 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:43.460 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2026-02-17T05:06:42.356497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001735"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084394,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:43.460 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:43.460 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80061, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:31.820443+0000', 'last_change': '2026-02-17T05:06:31.820443+0000', 'last_active': '2026-02-17T05:06:31.820443+0000', 'last_peered': '2026-02-17T05:06:31.820443+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:31.820443+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:43.460 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:43.676 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:43.676 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":770,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:43.684 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:43.685 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:43.865 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:43.865 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:43.875 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2026-02-17T05:06:42.356497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001735"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084394,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:46.877 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:47.055 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:47.056 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:47.064 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2026-02-17T05:06:46.357008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001801"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084395,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:47.065 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:47.240 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:47.240 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:47.249 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2026-02-17T05:06:46.357008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001801"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084395,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:47.249 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:47.249 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80061, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:31.820443+0000', 'last_change': '2026-02-17T05:06:31.820443+0000', 'last_active': '2026-02-17T05:06:31.820443+0000', 'last_peered': '2026-02-17T05:06:31.820443+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:31.820443+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:47.249 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:47.465 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:47.465 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":774,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:47.474 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:47.475 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:47.650 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:47.651 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:47.659 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2026-02-17T05:06:46.357008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001801"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084395,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:50.661 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:50.841 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:50.841 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:50.849 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2026-02-17T05:06:50.357568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001592"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084396,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:50.851 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:51.028 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:51.028 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:51.036 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2026-02-17T05:06:50.357568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001592"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084396,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:51.037 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:51.037 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80061, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:31.820443+0000', 'last_change': '2026-02-17T05:06:31.820443+0000', 'last_active': '2026-02-17T05:06:31.820443+0000', 'last_peered': '2026-02-17T05:06:31.820443+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:31.820443+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:51.037 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:51.264 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:51.264 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":778,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:51.274 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:51.274 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:51.451 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:51.451 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:51.460 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2026-02-17T05:06:50.357568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001592"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084396,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:54.461 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:54.637 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:54.638 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:54.646 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2026-02-17T05:06:54.358162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001664"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084397,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:54.647 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:54.821 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:54.821 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:54.830 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2026-02-17T05:06:54.358162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001664"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084397,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:54.831 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:54.831 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80061, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:31.820443+0000', 'last_change': '2026-02-17T05:06:31.820443+0000', 'last_active': '2026-02-17T05:06:31.820443+0000', 'last_peered': '2026-02-17T05:06:31.820443+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:31.820443+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:54.832 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:55.059 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:55.060 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":781,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:55.070 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:55.070 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:55.245 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:55.246 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:55.254 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2026-02-17T05:06:54.358162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001664"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084397,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:58.256 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:58.431 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:58.431 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:58.440 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2026-02-17T05:06:58.358639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001631"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084397,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:58.441 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:58.616 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:58.617 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:58.624 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2026-02-17T05:06:58.358639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001631"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084397,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:06:58.625 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:06:58.625 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80061, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:31.820443+0000', 'last_change': '2026-02-17T05:06:31.820443+0000', 'last_active': '2026-02-17T05:06:31.820443+0000', 'last_peered': '2026-02-17T05:06:31.820443+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:31.820443+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:06:58.625 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:06:58.844 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:58.844 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":785,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:06:58.854 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:06:58.855 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:06:59.027 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:06:59.027 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:06:59.035 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2026-02-17T05:06:58.358639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001631"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084397,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:02.035 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:02.213 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:02.214 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:02.222 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2026-02-17T05:07:00.358889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001619"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084398,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:02.223 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:02.396 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:02.397 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:02.406 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2026-02-17T05:07:02.359184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001616"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084398,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:02.406 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:02.406 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80061, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:06:31.820443+0000', 'last_change': '2026-02-17T05:06:31.820443+0000', 'last_active': '2026-02-17T05:06:31.820443+0000', 'last_peered': '2026-02-17T05:06:31.820443+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:06:31.820443+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:02.406 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:02.620 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:02.620 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":789,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:07:02.629 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:02.629 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:02.804 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:02.805 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:02.813 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2026-02-17T05:07:02.359184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001616"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80061,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:06:31.820443+0000","last_change":"2026-02-17T05:06:31.820443+0000","last_active":"2026-02-17T05:06:31.820443+0000","last_peered":"2026-02-17T05:06:31.820443+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:06:31.820443+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084398,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:05.815 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:05.991 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:05.992 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:06.000 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2026-02-17T05:07:04.359438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001595"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084399,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:06.001 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:06.174 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:06.175 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:06.182 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2026-02-17T05:07:04.359438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001595"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084399,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:06.183 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:06.183 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80065, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:01.821003+0000', 'last_change': '2026-02-17T05:07:01.821003+0000', 'last_active': '2026-02-17T05:07:01.821003+0000', 'last_peered': '2026-02-17T05:07:01.821003+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:01.821003+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:06.183 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:06.399 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:06.399 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":793,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:07:06.409 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:06.409 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:06.585 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:06.586 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:06.594 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2026-02-17T05:07:06.359647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001485"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084399,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:09.595 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:09.769 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:09.770 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:09.778 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2026-02-17T05:07:08.359883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001461"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084399,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:09.779 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:09.954 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:09.954 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:09.964 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2026-02-17T05:07:08.359883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001461"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084399,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:09.965 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:09.965 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80065, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:01.821003+0000', 'last_change': '2026-02-17T05:07:01.821003+0000', 'last_active': '2026-02-17T05:07:01.821003+0000', 'last_peered': '2026-02-17T05:07:01.821003+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:01.821003+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:09.965 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:10.186 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:10.186 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":797,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:07:10.196 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:10.196 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:10.377 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:10.377 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:10.387 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2026-02-17T05:07:10.360174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001534"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084400,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:13.388 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:13.570 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:13.571 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:13.580 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2026-02-17T05:07:12.360422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001532"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084400,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:13.582 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:13.758 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:13.758 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:13.766 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2026-02-17T05:07:12.360422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001532"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084400,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:13.767 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:13.767 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80065, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:01.821003+0000', 'last_change': '2026-02-17T05:07:01.821003+0000', 'last_active': '2026-02-17T05:07:01.821003+0000', 'last_peered': '2026-02-17T05:07:01.821003+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:01.821003+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:13.767 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:13.983 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:13.983 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":800,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:07:13.992 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:13.992 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:14.171 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:14.171 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:14.181 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2026-02-17T05:07:12.360422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001532"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084400,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:17.182 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:17.362 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:17.362 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:17.372 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2026-02-17T05:07:16.360972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001533"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084401,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:17.373 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:17.546 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:17.546 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:17.555 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2026-02-17T05:07:16.360972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001533"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084401,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:17.555 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:17.555 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80065, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:01.821003+0000', 'last_change': '2026-02-17T05:07:01.821003+0000', 'last_active': '2026-02-17T05:07:01.821003+0000', 'last_peered': '2026-02-17T05:07:01.821003+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:01.821003+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:17.555 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:17.772 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:17.772 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":804,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":12,"modified":"2026-02-17T05:05:24.346810+0000","services":{}},"progress_events":{}} 2026-02-17T05:07:17.780 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:17.780 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:17.953 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:17.953 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:17.961 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2026-02-17T05:07:16.360972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001533"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084401,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:20.962 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:21.139 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:21.139 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:21.147 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2026-02-17T05:07:20.361580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001696"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084402,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:21.148 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:21.321 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:21.321 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:21.330 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2026-02-17T05:07:20.361580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001696"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084402,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:21.331 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:21.331 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80065, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:01.821003+0000', 'last_change': '2026-02-17T05:07:01.821003+0000', 'last_active': '2026-02-17T05:07:01.821003+0000', 'last_peered': '2026-02-17T05:07:01.821003+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:01.821003+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:21.331 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:21.549 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:21.549 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":808,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:07:21.560 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:21.561 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:21.829 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:07:21.828+0000 7f41077fe640 0 --2- 10.20.193.100:0/113712471 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7f410805c490 0x7f410810f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:07:22.832 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:22.832 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:22.846 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2026-02-17T05:07:22.361883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084402,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:25.847 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:26.023 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:26.024 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:26.032 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2026-02-17T05:07:24.362128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001706"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084403,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:26.033 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:26.214 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:26.214 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:26.223 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2026-02-17T05:07:24.362128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001706"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084403,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:26.224 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:26.224 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80065, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:01.821003+0000', 'last_change': '2026-02-17T05:07:01.821003+0000', 'last_active': '2026-02-17T05:07:01.821003+0000', 'last_peered': '2026-02-17T05:07:01.821003+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:01.821003+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:26.224 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:26.441 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:26.441 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":813,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:07:26.450 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:26.451 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:26.628 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:26.629 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:26.638 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2026-02-17T05:07:26.362429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001770"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084403,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:29.640 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:29.818 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:29.818 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:29.827 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2026-02-17T05:07:28.362650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001678"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084403,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:29.829 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:30.005 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:30.005 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:30.014 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2026-02-17T05:07:28.362650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001678"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084403,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:30.015 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:30.015 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80065, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:01.821003+0000', 'last_change': '2026-02-17T05:07:01.821003+0000', 'last_active': '2026-02-17T05:07:01.821003+0000', 'last_peered': '2026-02-17T05:07:01.821003+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:01.821003+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:30.015 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:30.238 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:30.238 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":817,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:07:30.246 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:30.247 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:30.422 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:30.423 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:30.431 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2026-02-17T05:07:30.362883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001634"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:33.433 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:33.610 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:33.610 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:33.619 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2026-02-17T05:07:32.363144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001564"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:33.620 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:33.795 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:33.795 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:33.804 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2026-02-17T05:07:32.363144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001564"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:33.805 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:33.805 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80065, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:01.821003+0000', 'last_change': '2026-02-17T05:07:01.821003+0000', 'last_active': '2026-02-17T05:07:01.821003+0000', 'last_peered': '2026-02-17T05:07:01.821003+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:01.821003+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:33.805 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:34.024 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:34.024 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":820,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:07:34.034 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:34.034 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:34.213 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:34.213 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:34.223 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2026-02-17T05:07:32.363144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001564"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80065,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:01.821003+0000","last_change":"2026-02-17T05:07:01.821003+0000","last_active":"2026-02-17T05:07:01.821003+0000","last_peered":"2026-02-17T05:07:01.821003+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:01.821003+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:37.225 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:37.402 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:37.403 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:37.411 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2026-02-17T05:07:36.363743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001614"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084405,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:37.413 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:37.587 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:37.588 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:37.596 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2026-02-17T05:07:36.363743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001614"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084405,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:37.597 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:37.597 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80069, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:31.821671+0000', 'last_change': '2026-02-17T05:07:31.821671+0000', 'last_active': '2026-02-17T05:07:31.821671+0000', 'last_peered': '2026-02-17T05:07:31.821671+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:31.821671+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:37.598 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:37.816 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:37.816 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":824,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:07:37.825 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:37.825 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:37.999 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:37.999 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:38.008 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2026-02-17T05:07:36.363743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001614"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084405,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:41.009 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:41.186 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:41.186 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:41.196 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2026-02-17T05:07:40.364188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001537"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084406,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:41.197 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:41.373 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:41.374 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:41.382 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2026-02-17T05:07:40.364188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001537"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084406,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:41.382 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:41.382 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80069, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:31.821671+0000', 'last_change': '2026-02-17T05:07:31.821671+0000', 'last_active': '2026-02-17T05:07:31.821671+0000', 'last_peered': '2026-02-17T05:07:31.821671+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:31.821671+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:41.383 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:41.602 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:41.602 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":828,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:07:41.611 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:41.611 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:41.784 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:41.785 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:41.793 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2026-02-17T05:07:40.364188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001537"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084406,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:44.795 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:44.971 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:44.972 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:44.980 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2026-02-17T05:07:44.364673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001529"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084407,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:44.982 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:45.157 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:45.158 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:45.166 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2026-02-17T05:07:44.364673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001529"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084407,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:45.167 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:45.168 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80069, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:31.821671+0000', 'last_change': '2026-02-17T05:07:31.821671+0000', 'last_active': '2026-02-17T05:07:31.821671+0000', 'last_peered': '2026-02-17T05:07:31.821671+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:31.821671+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:45.168 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:45.385 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:45.385 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":832,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:07:45.393 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:45.394 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:45.568 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:45.569 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:45.577 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2026-02-17T05:07:44.364673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001529"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084407,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:48.579 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:48.755 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:48.755 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:48.764 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2026-02-17T05:07:48.365245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001502"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084407,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:48.766 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:48.948 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:48.948 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:48.958 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2026-02-17T05:07:48.365245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001502"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084407,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:48.958 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:48.958 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80069, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:31.821671+0000', 'last_change': '2026-02-17T05:07:31.821671+0000', 'last_active': '2026-02-17T05:07:31.821671+0000', 'last_peered': '2026-02-17T05:07:31.821671+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:31.821671+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:48.959 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:49.178 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:49.178 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":836,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:07:49.187 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:49.188 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:49.362 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:49.363 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:49.371 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2026-02-17T05:07:48.365245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001502"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084407,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:52.372 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:52.550 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:52.550 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:52.559 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2026-02-17T05:07:52.365812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001623"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084408,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:52.560 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:52.735 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:52.736 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:52.745 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2026-02-17T05:07:52.365812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001623"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084408,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:52.746 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:52.746 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80069, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:31.821671+0000', 'last_change': '2026-02-17T05:07:31.821671+0000', 'last_active': '2026-02-17T05:07:31.821671+0000', 'last_peered': '2026-02-17T05:07:31.821671+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:31.821671+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:52.747 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:52.963 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:52.964 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":839,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:07:52.973 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:52.973 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:53.151 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:53.151 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:53.160 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2026-02-17T05:07:52.365812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001623"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084408,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:56.161 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:56.341 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:56.341 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:56.350 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2026-02-17T05:07:54.366051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001607"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084409,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:56.352 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:56.528 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:56.528 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:56.538 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2026-02-17T05:07:56.366295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001621"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084409,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:56.538 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:07:56.538 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80069, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:31.821671+0000', 'last_change': '2026-02-17T05:07:31.821671+0000', 'last_active': '2026-02-17T05:07:31.821671+0000', 'last_peered': '2026-02-17T05:07:31.821671+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:31.821671+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:07:56.539 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:07:56.762 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:56.762 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":843,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:07:56.770 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:07:56.770 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:07:56.945 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:07:56.946 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:07:56.956 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2026-02-17T05:07:56.366295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001621"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084409,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705838,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:07:59.957 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:00.141 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:00.141 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:00.150 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2026-02-17T05:07:58.366505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001546"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084409,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:00.151 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:00.327 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:00.327 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:00.335 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2026-02-17T05:07:58.366505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001546"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:03:58.473241+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:03:58.473241+0000","last_peered":"2026-02-17T05:03:58.473241+0000","last_clean":"2026-02-17T05:03:58.473241+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:03:58.473241+0000","last_undegraded":"2026-02-17T05:03:58.473241+0000","last_fullsized":"2026-02-17T05:03:58.473241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084409,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:00.336 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:00.337 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80069, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:31.821671+0000', 'last_change': '2026-02-17T05:07:31.821671+0000', 'last_active': '2026-02-17T05:07:31.821671+0000', 'last_peered': '2026-02-17T05:07:31.821671+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:31.821671+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:00.337 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:00.556 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:00.556 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":847,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:08:00.566 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:00.567 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:00.741 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:00.742 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:00.750 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2026-02-17T05:08:00.366797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084410,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:03.751 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:03.931 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:03.931 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:03.940 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2026-02-17T05:08:02.367044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084410,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:03.941 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:04.117 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:04.118 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:04.126 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2026-02-17T05:08:02.367044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80069,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:07:31.821671+0000","last_change":"2026-02-17T05:07:31.821671+0000","last_active":"2026-02-17T05:07:31.821671+0000","last_peered":"2026-02-17T05:07:31.821671+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:07:31.821671+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084410,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:04.126 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:04.126 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80069, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:07:31.821671+0000', 'last_change': '2026-02-17T05:07:31.821671+0000', 'last_active': '2026-02-17T05:07:31.821671+0000', 'last_peered': '2026-02-17T05:07:31.821671+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:07:31.821671+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:04.126 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:04.345 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:04.345 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":851,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:08:04.354 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:04.354 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:04.529 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:04.529 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:04.538 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2026-02-17T05:08:04.367282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001470"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80073,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822145+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822145+0000","last_peered":"2026-02-17T05:08:01.822145+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822145+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084411,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:07.540 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:07.717 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:07.718 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:07.726 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2026-02-17T05:08:06.367547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001495"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80073,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822145+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822145+0000","last_peered":"2026-02-17T05:08:01.822145+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822145+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084411,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:07.727 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:07.908 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:07.908 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:07.916 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2026-02-17T05:08:06.367547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001495"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80073,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822145+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822145+0000","last_peered":"2026-02-17T05:08:01.822145+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822145+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084411,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:07.917 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:07.917 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80073, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:01.822145+0000', 'last_change': '2026-02-17T05:08:01.822145+0000', 'last_active': '2026-02-17T05:08:01.822145+0000', 'last_peered': '2026-02-17T05:08:01.822145+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:01.822145+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:07.917 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:08.134 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:08.134 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":855,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:08:08.143 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:08.143 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:08.413 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:08.414 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:08.422 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2026-02-17T05:08:08.367834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001539"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80073,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822145+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822145+0000","last_peered":"2026-02-17T05:08:01.822145+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822145+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084411,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:11.424 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:11.600 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:11.600 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:11.608 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2026-02-17T05:08:10.368061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80073,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822145+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822145+0000","last_peered":"2026-02-17T05:08:01.822145+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822145+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084412,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:11.609 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:11.785 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:11.786 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:11.795 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2026-02-17T05:08:10.368061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80073,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822145+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822145+0000","last_peered":"2026-02-17T05:08:01.822145+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822145+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084412,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:11.797 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:11.797 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80073, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:01.822145+0000', 'last_change': '2026-02-17T05:08:01.822145+0000', 'last_active': '2026-02-17T05:08:01.822145+0000', 'last_peered': '2026-02-17T05:08:01.822145+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:01.822145+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:11.798 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:12.020 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:12.020 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":858,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":13,"modified":"2026-02-17T05:07:18.361298+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:08:12.029 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:12.029 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:12.207 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:12.208 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:12.216 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2026-02-17T05:08:10.368061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80073,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822145+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822145+0000","last_peered":"2026-02-17T05:08:01.822145+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822145+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084412,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:15.217 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:15.394 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:15.394 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:15.402 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2026-02-17T05:08:14.368570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084413,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:15.403 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:15.579 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:15.579 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:15.588 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2026-02-17T05:08:14.368570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084413,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:15.589 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:15.590 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80074, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:01.822170+0000', 'last_change': '2026-02-17T05:08:01.822145+0000', 'last_active': '2026-02-17T05:08:01.822170+0000', 'last_peered': '2026-02-17T05:08:01.822170+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:01.822170+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:15.590 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:15.809 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:15.810 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":862,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:15.819 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:15.819 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:15.997 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:15.997 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:16.008 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2026-02-17T05:08:14.368570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084413,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:19.009 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:19.189 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:19.189 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:19.200 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2026-02-17T05:08:18.369057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001510"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084413,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:19.201 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:19.380 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:19.380 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:19.389 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2026-02-17T05:08:18.369057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001510"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084413,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:19.391 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:19.391 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80074, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:01.822170+0000', 'last_change': '2026-02-17T05:08:01.822145+0000', 'last_active': '2026-02-17T05:08:01.822170+0000', 'last_peered': '2026-02-17T05:08:01.822170+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:01.822170+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:19.391 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:19.659 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:08:19.658+0000 7ff20d039640 0 --2- 10.20.193.100:0/1920767045 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7ff208137860 0x7ff2081993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:08:19.811 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:19.811 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":866,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:19.821 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:19.821 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:19.998 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:19.998 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:20.008 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2026-02-17T05:08:18.369057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001510"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084413,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:23.010 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:23.189 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:23.190 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:23.198 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2026-02-17T05:08:22.369526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001464"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084414,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:23.200 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:23.377 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:23.377 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:23.385 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2026-02-17T05:08:22.369526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001464"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084414,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:23.386 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:23.386 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80074, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:01.822170+0000', 'last_change': '2026-02-17T05:08:01.822145+0000', 'last_active': '2026-02-17T05:08:01.822170+0000', 'last_peered': '2026-02-17T05:08:01.822170+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:01.822170+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:23.386 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:23.606 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:23.606 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":870,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:23.616 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:23.617 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:23.797 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:23.798 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:23.806 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2026-02-17T05:08:22.369526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001464"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084414,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:26.808 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:26.987 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:26.988 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:26.997 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2026-02-17T05:08:26.370058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001487"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084415,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:26.998 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:27.175 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:27.175 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:27.184 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2026-02-17T05:08:26.370058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001487"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084415,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:27.185 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:27.186 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80074, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:01.822170+0000', 'last_change': '2026-02-17T05:08:01.822145+0000', 'last_active': '2026-02-17T05:08:01.822170+0000', 'last_peered': '2026-02-17T05:08:01.822170+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:01.822170+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:27.186 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:27.412 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:27.412 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":874,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:27.422 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:27.422 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:27.595 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:27.595 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:27.603 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2026-02-17T05:08:26.370058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001487"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084415,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705844,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:30.604 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:30.781 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:30.781 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:30.789 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2026-02-17T05:08:30.370551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084416,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:30.790 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:30.964 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:30.964 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:30.972 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2026-02-17T05:08:30.370551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084416,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:30.973 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:30.973 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80074, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:01.822170+0000', 'last_change': '2026-02-17T05:08:01.822145+0000', 'last_active': '2026-02-17T05:08:01.822170+0000', 'last_peered': '2026-02-17T05:08:01.822170+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:01.822170+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:30.973 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:31.189 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:31.190 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":878,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:31.198 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:31.198 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:31.373 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:31.374 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:31.382 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2026-02-17T05:08:30.370551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80074,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:01.822170+0000","last_change":"2026-02-17T05:08:01.822145+0000","last_active":"2026-02-17T05:08:01.822170+0000","last_peered":"2026-02-17T05:08:01.822170+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:01.822170+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084416,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:34.384 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:34.559 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:34.559 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:34.568 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2026-02-17T05:08:34.371031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001505"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084417,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:34.568 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:34.743 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:34.743 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:34.752 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2026-02-17T05:08:34.371031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001505"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084417,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:34.753 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:34.753 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80078, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:31.822766+0000', 'last_change': '2026-02-17T05:08:31.822766+0000', 'last_active': '2026-02-17T05:08:31.822766+0000', 'last_peered': '2026-02-17T05:08:31.822766+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:31.822766+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:34.754 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:34.977 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:34.977 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":881,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:34.987 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:34.987 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:35.164 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:35.164 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:35.173 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2026-02-17T05:08:34.371031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001505"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084417,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:38.175 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:38.349 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:38.350 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:38.358 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2026-02-17T05:08:36.371321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001543"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084417,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:38.359 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:38.532 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:38.533 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:38.541 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2026-02-17T05:08:38.371594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001536"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084417,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:38.542 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:38.542 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80078, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:31.822766+0000', 'last_change': '2026-02-17T05:08:31.822766+0000', 'last_active': '2026-02-17T05:08:31.822766+0000', 'last_peered': '2026-02-17T05:08:31.822766+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:31.822766+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:38.543 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:38.765 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:38.765 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":885,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:38.774 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:38.774 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:38.956 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:38.956 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:38.966 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2026-02-17T05:08:38.371594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001536"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084417,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:41.969 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:42.144 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:42.144 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:42.153 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2026-02-17T05:08:40.371803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001527"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084418,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:42.154 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:42.326 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:42.326 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:42.334 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2026-02-17T05:08:40.371803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001527"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084418,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:42.335 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:42.335 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80078, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:31.822766+0000', 'last_change': '2026-02-17T05:08:31.822766+0000', 'last_active': '2026-02-17T05:08:31.822766+0000', 'last_peered': '2026-02-17T05:08:31.822766+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:31.822766+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:42.335 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:42.553 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:42.553 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":889,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:42.562 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:42.563 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:42.735 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:42.735 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:42.743 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2026-02-17T05:08:42.372047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001496"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084418,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:45.745 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:45.926 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:45.926 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:45.934 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2026-02-17T05:08:44.372316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:45.936 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:46.110 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:46.110 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:46.119 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2026-02-17T05:08:44.372316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:46.120 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:46.120 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80078, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:31.822766+0000', 'last_change': '2026-02-17T05:08:31.822766+0000', 'last_active': '2026-02-17T05:08:31.822766+0000', 'last_peered': '2026-02-17T05:08:31.822766+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:31.822766+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:46.120 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:46.333 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:46.333 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":893,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:46.342 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:46.342 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:46.521 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:46.521 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:46.530 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2026-02-17T05:08:46.372637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:49.532 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:49.709 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:49.709 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:49.717 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2026-02-17T05:08:48.372883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:49.718 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:49.892 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:49.892 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:49.901 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2026-02-17T05:08:48.372883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:49.901 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:49.901 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80078, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:31.822766+0000', 'last_change': '2026-02-17T05:08:31.822766+0000', 'last_active': '2026-02-17T05:08:31.822766+0000', 'last_peered': '2026-02-17T05:08:31.822766+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:31.822766+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:49.901 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:50.119 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:50.119 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":897,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:50.129 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:50.129 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:50.307 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:50.307 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:50.316 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2026-02-17T05:08:48.372883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:53.317 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:53.500 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:53.500 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:53.509 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2026-02-17T05:08:52.373393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084420,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:53.511 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:53.686 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:53.686 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:53.695 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2026-02-17T05:08:52.373393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084420,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:53.695 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:53.695 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80078, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:31.822766+0000', 'last_change': '2026-02-17T05:08:31.822766+0000', 'last_active': '2026-02-17T05:08:31.822766+0000', 'last_peered': '2026-02-17T05:08:31.822766+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:31.822766+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:53.695 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:53.915 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:53.915 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":900,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:53.926 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:53.926 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:54.101 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:54.102 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:54.110 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2026-02-17T05:08:52.373393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084420,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:57.112 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:57.287 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:57.288 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:57.297 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2026-02-17T05:08:56.373965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001648"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084421,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:57.298 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:57.470 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:57.470 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:57.479 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2026-02-17T05:08:56.373965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001648"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084421,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:08:57.479 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:08:57.479 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80078, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:31.822766+0000', 'last_change': '2026-02-17T05:08:31.822766+0000', 'last_active': '2026-02-17T05:08:31.822766+0000', 'last_peered': '2026-02-17T05:08:31.822766+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:31.822766+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:08:57.480 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:08:57.703 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:57.704 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":904,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:08:57.712 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:08:57.713 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:08:57.887 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:08:57.888 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:08:57.896 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2026-02-17T05:08:56.373965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001648"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084421,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:00.898 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:01.073 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:01.074 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:01.082 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2026-02-17T05:09:00.374363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001480"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084422,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:01.082 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:01.351 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:09:01.350+0000 7f4a89dcf640 0 --2- 10.20.193.100:0/536030934 >> [v2:10.20.193.100:3300/0,v1:10.20.193.100:6789/0] conn(0x7f4a84156150 0x7f4a8417bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:09:01.459 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:01.459 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:01.468 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2026-02-17T05:09:00.374363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001480"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084422,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:01.469 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:01.469 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80078, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:08:31.822766+0000', 'last_change': '2026-02-17T05:08:31.822766+0000', 'last_active': '2026-02-17T05:08:31.822766+0000', 'last_peered': '2026-02-17T05:08:31.822766+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:08:31.822766+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:01.469 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:01.686 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:01.686 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":908,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:01.695 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:01.695 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:01.873 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:01.874 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:01.882 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2026-02-17T05:09:00.374363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001480"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80078,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:08:31.822766+0000","last_change":"2026-02-17T05:08:31.822766+0000","last_active":"2026-02-17T05:08:31.822766+0000","last_peered":"2026-02-17T05:08:31.822766+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:08:31.822766+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084422,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:04.884 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:05.064 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:05.065 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:05.073 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2026-02-17T05:09:04.374849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084423,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:05.074 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:05.250 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:05.250 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:05.259 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2026-02-17T05:09:04.374849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084423,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:05.260 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:05.260 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80082, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:01.823298+0000', 'last_change': '2026-02-17T05:09:01.823298+0000', 'last_active': '2026-02-17T05:09:01.823298+0000', 'last_peered': '2026-02-17T05:09:01.823298+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:01.823298+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:05.260 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:05.484 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:05.484 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":912,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:05.494 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:05.494 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:05.672 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:05.673 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:05.681 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2026-02-17T05:09:04.374849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084423,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:08.682 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:08.860 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:08.860 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:08.869 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2026-02-17T05:09:08.375309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001344"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084423,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:08.870 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:09.048 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:09.048 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:09.057 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2026-02-17T05:09:08.375309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001344"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084423,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:09.058 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:09.058 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80082, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:01.823298+0000', 'last_change': '2026-02-17T05:09:01.823298+0000', 'last_active': '2026-02-17T05:09:01.823298+0000', 'last_peered': '2026-02-17T05:09:01.823298+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:01.823298+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:09.058 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:09.279 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:09.279 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":916,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:09.288 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:09.289 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:09.463 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:09.464 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:09.472 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2026-02-17T05:09:08.375309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001344"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084423,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:12.475 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:12.650 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:12.651 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:12.660 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2026-02-17T05:09:12.375904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084424,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:12.661 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:12.834 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:12.835 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:12.843 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2026-02-17T05:09:12.375904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084424,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:12.844 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:12.844 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80082, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:01.823298+0000', 'last_change': '2026-02-17T05:09:01.823298+0000', 'last_active': '2026-02-17T05:09:01.823298+0000', 'last_peered': '2026-02-17T05:09:01.823298+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:01.823298+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:12.844 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:13.062 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:13.062 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":919,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:13.071 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:13.071 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:13.251 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:13.252 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:13.261 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2026-02-17T05:09:12.375904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084424,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:16.263 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:16.441 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:16.441 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:16.449 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2026-02-17T05:09:16.376350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001500"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084425,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:16.450 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:16.623 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:16.623 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:16.632 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2026-02-17T05:09:16.376350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001500"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084425,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:16.633 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:16.633 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80082, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:01.823298+0000', 'last_change': '2026-02-17T05:09:01.823298+0000', 'last_active': '2026-02-17T05:09:01.823298+0000', 'last_peered': '2026-02-17T05:09:01.823298+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:01.823298+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:16.633 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:16.850 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:16.850 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":923,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:16.859 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:16.859 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:17.039 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:17.039 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:17.048 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2026-02-17T05:09:16.376350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001500"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084425,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:20.051 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:20.228 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:20.228 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:20.236 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2026-02-17T05:09:18.376590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001491"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084425,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:20.237 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:20.412 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:20.413 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:20.421 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2026-02-17T05:09:20.376862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001552"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084426,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:20.421 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:20.421 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80082, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:01.823298+0000', 'last_change': '2026-02-17T05:09:01.823298+0000', 'last_active': '2026-02-17T05:09:01.823298+0000', 'last_peered': '2026-02-17T05:09:01.823298+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:01.823298+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:20.421 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:20.637 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:20.637 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":927,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:20.646 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:20.647 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:20.823 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:20.823 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:20.831 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2026-02-17T05:09:20.376862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001552"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084426,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:23.833 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:24.007 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:24.007 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:24.015 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2026-02-17T05:09:22.377123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001533"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084426,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:24.016 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:24.189 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:24.190 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:24.199 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2026-02-17T05:09:22.377123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001533"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084426,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:24.199 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:24.199 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80082, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:01.823298+0000', 'last_change': '2026-02-17T05:09:01.823298+0000', 'last_active': '2026-02-17T05:09:01.823298+0000', 'last_peered': '2026-02-17T05:09:01.823298+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:01.823298+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:24.200 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:24.419 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:24.419 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":931,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:24.429 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:24.429 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:24.608 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:24.609 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:24.617 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2026-02-17T05:09:24.377409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001504"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:27.620 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:27.798 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:27.798 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:27.807 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2026-02-17T05:09:26.377690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001578"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:27.808 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:28.077 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:09:28.076+0000 7f42c0b09640 0 --2- 10.20.193.100:0/2409814295 >> [v2:10.20.193.100:3302/0,v1:10.20.193.100:6791/0] conn(0x7f42bc07e8b0 0x7f42bc124320 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:09:28.186 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:28.186 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:28.195 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2026-02-17T05:09:26.377690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001578"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:28.196 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:28.196 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80082, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:01.823298+0000', 'last_change': '2026-02-17T05:09:01.823298+0000', 'last_active': '2026-02-17T05:09:01.823298+0000', 'last_peered': '2026-02-17T05:09:01.823298+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:01.823298+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:28.197 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:28.412 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:28.412 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":935,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:28.421 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:28.421 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:28.594 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:28.595 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:28.603 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2026-02-17T05:09:28.377999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001649"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:31.605 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:31.782 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:31.782 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:31.791 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2026-02-17T05:09:30.378291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084428,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:31.792 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:31.967 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:31.968 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:31.975 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2026-02-17T05:09:30.378291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084428,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:31.976 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:31.977 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80082, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:01.823298+0000', 'last_change': '2026-02-17T05:09:01.823298+0000', 'last_active': '2026-02-17T05:09:01.823298+0000', 'last_peered': '2026-02-17T05:09:01.823298+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:01.823298+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:31.977 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:32.194 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:32.194 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":939,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:32.204 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:32.204 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:32.379 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:32.379 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:32.387 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2026-02-17T05:09:30.378291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001701"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80082,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:01.823298+0000","last_change":"2026-02-17T05:09:01.823298+0000","last_active":"2026-02-17T05:09:01.823298+0000","last_peered":"2026-02-17T05:09:01.823298+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:01.823298+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084428,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:35.388 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:35.567 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:35.568 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:35.577 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2026-02-17T05:09:34.378990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001867"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084429,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:35.578 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:35.755 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:35.755 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:35.764 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2026-02-17T05:09:34.378990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001867"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084429,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:35.765 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:35.765 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80086, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:31.823811+0000', 'last_change': '2026-02-17T05:09:31.823811+0000', 'last_active': '2026-02-17T05:09:31.823811+0000', 'last_peered': '2026-02-17T05:09:31.823811+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:31.823811+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:35.765 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:35.980 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:35.980 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":942,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:35.989 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:35.989 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:36.165 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:36.165 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:36.174 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2026-02-17T05:09:34.378990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001867"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084429,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:39.176 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:39.355 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:39.356 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:39.364 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2026-02-17T05:09:38.379602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001912"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084429,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:39.365 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:39.635 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:09:39.634+0000 7f88c95d1640 0 --2- 10.20.193.100:0/727253465 >> [v2:10.20.193.100:3302/0,v1:10.20.193.100:6791/0] conn(0x7f88c4175690 0x7f88c4195a70 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=17138209323423541474 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:09:39.742 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:39.743 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:39.752 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2026-02-17T05:09:38.379602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001912"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084429,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:39.753 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:39.754 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80086, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:31.823811+0000', 'last_change': '2026-02-17T05:09:31.823811+0000', 'last_active': '2026-02-17T05:09:31.823811+0000', 'last_peered': '2026-02-17T05:09:31.823811+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:31.823811+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:39.754 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:39.970 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:39.970 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":946,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:39.979 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:39.979 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:40.157 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:40.157 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:40.166 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2026-02-17T05:09:38.379602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001912"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084429,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:43.168 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:43.350 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:43.351 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:43.361 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2026-02-17T05:09:42.380208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001916"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084430,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:43.361 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:43.536 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:43.537 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:43.545 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2026-02-17T05:09:42.380208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001916"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084430,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:43.547 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:43.547 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80086, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:31.823811+0000', 'last_change': '2026-02-17T05:09:31.823811+0000', 'last_active': '2026-02-17T05:09:31.823811+0000', 'last_peered': '2026-02-17T05:09:31.823811+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:31.823811+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:43.547 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:43.764 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:43.764 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":950,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:43.774 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:43.775 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:43.951 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:43.951 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:43.960 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2026-02-17T05:09:42.380208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001916"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084430,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:46.962 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:47.139 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:47.140 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:47.149 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2026-02-17T05:09:46.380781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001790"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084431,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:47.149 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:47.323 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:47.324 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:47.332 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2026-02-17T05:09:46.380781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001790"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084431,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:47.333 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:47.333 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80086, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:31.823811+0000', 'last_change': '2026-02-17T05:09:31.823811+0000', 'last_active': '2026-02-17T05:09:31.823811+0000', 'last_peered': '2026-02-17T05:09:31.823811+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:31.823811+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:47.333 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:47.554 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:47.554 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":954,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":14,"modified":"2026-02-17T05:08:14.368611+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:47.563 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:47.564 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:47.743 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:47.743 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:47.752 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2026-02-17T05:09:46.380781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001790"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084431,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:50.754 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:50.934 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:50.934 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:50.942 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2026-02-17T05:09:50.381332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001730"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084432,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:50.943 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:51.121 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:51.122 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:51.131 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2026-02-17T05:09:50.381332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001730"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084432,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:51.132 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:51.132 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80086, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:31.823811+0000', 'last_change': '2026-02-17T05:09:31.823811+0000', 'last_active': '2026-02-17T05:09:31.823811+0000', 'last_peered': '2026-02-17T05:09:31.823811+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:31.823811+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:51.133 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:51.348 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:51.348 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":958,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":15,"modified":"2026-02-17T05:09:50.381376+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:09:51.357 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:51.357 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:51.531 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:51.532 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:51.540 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2026-02-17T05:09:50.381332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001730"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084432,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:54.541 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:54.717 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:54.718 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:54.728 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2026-02-17T05:09:54.381861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001652"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084433,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:54.729 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:54.907 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:54.908 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:54.916 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2026-02-17T05:09:54.381861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001652"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084433,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:54.917 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:54.917 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80086, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:31.823811+0000', 'last_change': '2026-02-17T05:09:31.823811+0000', 'last_active': '2026-02-17T05:09:31.823811+0000', 'last_peered': '2026-02-17T05:09:31.823811+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:31.823811+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:54.918 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:55.135 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:55.135 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":962,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":15,"modified":"2026-02-17T05:09:50.381376+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:09:55.145 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:55.145 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:55.320 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:55.321 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:55.329 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2026-02-17T05:09:54.381861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001652"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084433,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:58.331 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:58.515 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:58.516 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:58.525 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2026-02-17T05:09:58.382374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084433,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:58.526 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:58.704 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:58.704 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:58.713 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2026-02-17T05:09:58.382374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084433,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:09:58.713 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:09:58.713 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80086, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:31.823811+0000', 'last_change': '2026-02-17T05:09:31.823811+0000', 'last_active': '2026-02-17T05:09:31.823811+0000', 'last_peered': '2026-02-17T05:09:31.823811+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:31.823811+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:09:58.713 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:09:58.937 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:58.937 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":965,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:09:58.946 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:09:58.946 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:09:59.124 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:09:59.125 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:09:59.132 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2026-02-17T05:09:58.382374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:05:58.481952+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:05:58.481952+0000","last_peered":"2026-02-17T05:05:58.481952+0000","last_clean":"2026-02-17T05:05:58.481952+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:05:58.481952+0000","last_undegraded":"2026-02-17T05:05:58.481952+0000","last_fullsized":"2026-02-17T05:05:58.481952+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084433,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:02.134 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:02.312 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:02.312 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:02.321 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2026-02-17T05:10:00.382661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001636"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084434,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:02.322 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:02.501 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:02.501 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:02.510 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2026-02-17T05:10:02.383011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001678"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084434,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:02.511 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:02.511 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80086, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:09:31.823811+0000', 'last_change': '2026-02-17T05:09:31.823811+0000', 'last_active': '2026-02-17T05:09:31.823811+0000', 'last_peered': '2026-02-17T05:09:31.823811+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:09:31.823811+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:02.512 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:02.731 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:02.731 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":969,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:02.740 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:02.741 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:02.915 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:02.916 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:02.924 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2026-02-17T05:10:02.383011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001678"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80086,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:09:31.823811+0000","last_change":"2026-02-17T05:09:31.823811+0000","last_active":"2026-02-17T05:09:31.823811+0000","last_peered":"2026-02-17T05:09:31.823811+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:09:31.823811+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084434,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:05.926 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:06.103 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:06.103 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:06.111 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2026-02-17T05:10:04.383292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80090,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824233+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824233+0000","last_peered":"2026-02-17T05:10:01.824233+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824233+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084435,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:06.112 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:06.289 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:06.290 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:06.298 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2026-02-17T05:10:04.383292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80090,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824233+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824233+0000","last_peered":"2026-02-17T05:10:01.824233+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824233+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084435,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:06.299 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:06.300 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80090, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:01.824233+0000', 'last_change': '2026-02-17T05:10:01.824233+0000', 'last_active': '2026-02-17T05:10:01.824233+0000', 'last_peered': '2026-02-17T05:10:01.824233+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:01.824233+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:06.300 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:06.518 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:06.518 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":973,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:06.527 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:06.527 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:06.701 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:06.701 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:06.709 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2026-02-17T05:10:06.383638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001777"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80090,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824233+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824233+0000","last_peered":"2026-02-17T05:10:01.824233+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824233+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084435,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:09.710 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:09.893 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:09.893 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:09.903 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2026-02-17T05:10:08.383841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80090,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824233+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824233+0000","last_peered":"2026-02-17T05:10:01.824233+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824233+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084435,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:09.904 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:10.078 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:10.079 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:10.087 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2026-02-17T05:10:08.383841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001702"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80090,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824233+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824233+0000","last_peered":"2026-02-17T05:10:01.824233+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824233+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084435,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:10.088 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:10.088 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80090, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:01.824233+0000', 'last_change': '2026-02-17T05:10:01.824233+0000', 'last_active': '2026-02-17T05:10:01.824233+0000', 'last_peered': '2026-02-17T05:10:01.824233+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:01.824233+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:10.088 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:10.305 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:10.305 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":977,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:10.313 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:10.314 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:10.488 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:10.488 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:10.497 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2026-02-17T05:10:10.384077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001703"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80090,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824233+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824233+0000","last_peered":"2026-02-17T05:10:01.824233+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824233+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084436,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:13.499 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:13.672 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:13.673 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:13.681 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2026-02-17T05:10:12.384313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001651"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80090,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824233+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824233+0000","last_peered":"2026-02-17T05:10:01.824233+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824233+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084436,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:13.682 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:13.858 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:13.858 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:13.867 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2026-02-17T05:10:12.384313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001651"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80090,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824233+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824233+0000","last_peered":"2026-02-17T05:10:01.824233+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824233+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084436,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:13.869 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:13.869 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80090, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:01.824233+0000', 'last_change': '2026-02-17T05:10:01.824233+0000', 'last_active': '2026-02-17T05:10:01.824233+0000', 'last_peered': '2026-02-17T05:10:01.824233+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:01.824233+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:13.869 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:14.088 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:14.088 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":981,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:14.098 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:14.099 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:14.274 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:14.274 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:14.282 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2026-02-17T05:10:12.384313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001651"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80090,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824233+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824233+0000","last_peered":"2026-02-17T05:10:01.824233+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824233+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084436,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:17.284 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:17.462 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:17.463 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:17.471 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2026-02-17T05:10:16.384752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001460"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084437,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:17.472 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:17.649 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:17.649 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:17.657 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2026-02-17T05:10:16.384752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001460"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084437,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:17.658 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:17.658 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80091, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:01.824277+0000', 'last_change': '2026-02-17T05:10:01.824233+0000', 'last_active': '2026-02-17T05:10:01.824277+0000', 'last_peered': '2026-02-17T05:10:01.824277+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:01.824277+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:17.659 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:17.878 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:17.878 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":984,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:17.887 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:17.888 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:18.064 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:18.064 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:18.073 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2026-02-17T05:10:16.384752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001460"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084437,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:21.075 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:21.251 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:21.251 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:21.260 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2026-02-17T05:10:20.385216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001374"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084438,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:21.261 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:21.443 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:21.443 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:21.452 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2026-02-17T05:10:20.385216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001374"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084438,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:21.453 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:21.453 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80091, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:01.824277+0000', 'last_change': '2026-02-17T05:10:01.824233+0000', 'last_active': '2026-02-17T05:10:01.824277+0000', 'last_peered': '2026-02-17T05:10:01.824277+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:01.824277+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:21.454 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:21.674 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:21.674 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":988,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:21.684 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:21.685 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:21.860 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:21.860 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:21.868 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2026-02-17T05:10:20.385216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001374"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084438,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:24.871 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:25.048 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:25.048 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:25.057 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2026-02-17T05:10:24.385701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001387"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084439,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:25.058 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:25.234 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:25.234 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:25.243 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2026-02-17T05:10:24.385701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001387"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084439,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:25.244 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:25.244 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80091, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:01.824277+0000', 'last_change': '2026-02-17T05:10:01.824233+0000', 'last_active': '2026-02-17T05:10:01.824277+0000', 'last_peered': '2026-02-17T05:10:01.824277+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:01.824277+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:25.244 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:25.465 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:25.466 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":992,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:25.476 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:25.476 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:25.652 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:25.652 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:25.662 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2026-02-17T05:10:24.385701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001387"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084439,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:28.663 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:28.838 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:28.839 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:28.848 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2026-02-17T05:10:28.386270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084439,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:28.849 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:29.028 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:29.028 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:29.038 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2026-02-17T05:10:28.386270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084439,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:29.038 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:29.039 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80091, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:01.824277+0000', 'last_change': '2026-02-17T05:10:01.824233+0000', 'last_active': '2026-02-17T05:10:01.824277+0000', 'last_peered': '2026-02-17T05:10:01.824277+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:01.824277+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:29.039 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:29.265 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:29.266 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":996,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:29.274 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:29.274 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:29.447 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:29.448 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:29.455 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2026-02-17T05:10:28.386270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084439,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:32.456 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:32.635 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:32.636 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:32.644 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2026-02-17T05:10:32.386827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001610"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084440,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:32.645 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:32.820 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:32.820 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:32.828 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2026-02-17T05:10:32.386827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001610"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084440,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:32.829 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:32.829 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80091, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:01.824277+0000', 'last_change': '2026-02-17T05:10:01.824233+0000', 'last_active': '2026-02-17T05:10:01.824277+0000', 'last_peered': '2026-02-17T05:10:01.824277+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:01.824277+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:32.829 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:33.051 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:33.051 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":999,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:33.062 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:33.062 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:33.238 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:33.239 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:33.247 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2026-02-17T05:10:32.386827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001610"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80091,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:01.824277+0000","last_change":"2026-02-17T05:10:01.824233+0000","last_active":"2026-02-17T05:10:01.824277+0000","last_peered":"2026-02-17T05:10:01.824277+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:01.824277+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084440,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:36.248 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:36.427 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:36.428 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:36.436 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2026-02-17T05:10:36.387387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001685"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084441,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:36.436 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:36.612 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:36.612 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:36.620 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2026-02-17T05:10:36.387387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001685"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084441,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:36.621 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:36.621 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80095, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:31.824830+0000', 'last_change': '2026-02-17T05:10:31.824830+0000', 'last_active': '2026-02-17T05:10:31.824830+0000', 'last_peered': '2026-02-17T05:10:31.824830+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:31.824830+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:36.621 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:36.838 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:36.839 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1003,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:36.847 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:36.848 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:37.026 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:37.026 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:37.037 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2026-02-17T05:10:36.387387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001685"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084441,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:40.038 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:40.311 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:40.312 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:40.321 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2026-02-17T05:10:38.387626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001644"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084441,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:40.322 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:40.499 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:40.500 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:40.509 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2026-02-17T05:10:40.387917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001646"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084442,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:40.510 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:40.510 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80095, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:31.824830+0000', 'last_change': '2026-02-17T05:10:31.824830+0000', 'last_active': '2026-02-17T05:10:31.824830+0000', 'last_peered': '2026-02-17T05:10:31.824830+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:31.824830+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:40.510 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:40.728 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:40.728 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1007,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:40.736 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:40.736 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:40.909 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:40.909 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:40.918 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2026-02-17T05:10:40.387917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001646"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084442,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:43.919 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:44.096 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:44.097 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:44.105 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2026-02-17T05:10:42.388126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001579"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084442,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:44.106 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:44.284 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:44.284 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:44.293 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2026-02-17T05:10:42.388126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001579"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084442,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:44.294 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:44.294 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80095, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:31.824830+0000', 'last_change': '2026-02-17T05:10:31.824830+0000', 'last_active': '2026-02-17T05:10:31.824830+0000', 'last_peered': '2026-02-17T05:10:31.824830+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:31.824830+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:44.295 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:44.516 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:44.516 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1011,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:44.526 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:44.526 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:44.705 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:44.705 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:44.714 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":530,"stamp":"2026-02-17T05:10:44.388410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001583"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:47.716 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:47.894 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:47.895 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:47.903 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2026-02-17T05:10:46.388663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:47.905 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:48.082 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:48.083 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:48.092 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2026-02-17T05:10:46.388663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:48.092 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:48.093 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80095, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:31.824830+0000', 'last_change': '2026-02-17T05:10:31.824830+0000', 'last_active': '2026-02-17T05:10:31.824830+0000', 'last_peered': '2026-02-17T05:10:31.824830+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:31.824830+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:48.093 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:48.318 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:48.318 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1015,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:48.327 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:48.328 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:48.503 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:48.504 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:48.512 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":532,"stamp":"2026-02-17T05:10:48.388887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001500"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:51.514 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:51.692 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:51.693 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:51.701 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2026-02-17T05:10:50.389128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001501"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084444,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:51.703 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:51.880 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:51.880 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:51.889 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2026-02-17T05:10:50.389128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001501"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084444,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:51.890 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:51.890 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80095, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:31.824830+0000', 'last_change': '2026-02-17T05:10:31.824830+0000', 'last_active': '2026-02-17T05:10:31.824830+0000', 'last_peered': '2026-02-17T05:10:31.824830+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:31.824830+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:51.890 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:52.111 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:52.111 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1019,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:52.119 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:52.119 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:52.294 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:52.294 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:52.303 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2026-02-17T05:10:50.389128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001501"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084444,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:55.304 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:55.481 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:55.482 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:55.490 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2026-02-17T05:10:54.389702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001576"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:55.491 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:55.670 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:55.670 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:55.678 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2026-02-17T05:10:54.389702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001576"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:55.679 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:55.679 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80095, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:31.824830+0000', 'last_change': '2026-02-17T05:10:31.824830+0000', 'last_active': '2026-02-17T05:10:31.824830+0000', 'last_peered': '2026-02-17T05:10:31.824830+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:31.824830+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:55.680 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:55.897 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:55.897 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1022,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:55.907 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:55.907 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:56.088 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:56.089 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:56.098 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2026-02-17T05:10:54.389702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001576"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:59.100 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:59.275 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:59.276 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:59.284 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":537,"stamp":"2026-02-17T05:10:58.390237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001574"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:59.286 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:59.461 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:59.461 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:59.469 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":537,"stamp":"2026-02-17T05:10:58.390237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001574"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:10:59.470 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:10:59.470 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80095, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:31.824830+0000', 'last_change': '2026-02-17T05:10:31.824830+0000', 'last_active': '2026-02-17T05:10:31.824830+0000', 'last_peered': '2026-02-17T05:10:31.824830+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:31.824830+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:10:59.471 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:10:59.690 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:59.691 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1026,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:10:59.699 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:10:59.699 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:10:59.876 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:10:59.876 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:10:59.885 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":537,"stamp":"2026-02-17T05:10:58.390237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001574"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:02.886 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:03.066 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:03.066 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:03.075 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":539,"stamp":"2026-02-17T05:11:02.390809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001680"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:03.076 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:03.249 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:03.249 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:03.257 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":539,"stamp":"2026-02-17T05:11:02.390809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001680"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:03.258 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:03.259 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80095, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:10:31.824830+0000', 'last_change': '2026-02-17T05:10:31.824830+0000', 'last_active': '2026-02-17T05:10:31.824830+0000', 'last_peered': '2026-02-17T05:10:31.824830+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:10:31.824830+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:03.259 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:03.478 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:03.478 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1030,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:03.488 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:03.488 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:03.665 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:03.665 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:03.673 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":539,"stamp":"2026-02-17T05:11:02.390809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001680"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80095,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:10:31.824830+0000","last_change":"2026-02-17T05:10:31.824830+0000","last_active":"2026-02-17T05:10:31.824830+0000","last_peered":"2026-02-17T05:10:31.824830+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:10:31.824830+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:06.675 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:06.946 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:11:06.945+0000 7fdf88e80640 0 --2- 10.20.193.100:0/950213596 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7fdf8405c490 0x7fdf84130550 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:11:07.056 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:07.056 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:07.065 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":541,"stamp":"2026-02-17T05:11:06.391379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001676"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:07.066 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:07.238 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:07.239 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:07.247 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":541,"stamp":"2026-02-17T05:11:06.391379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001676"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:07.248 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:07.248 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80099, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:01.825401+0000', 'last_change': '2026-02-17T05:11:01.825401+0000', 'last_active': '2026-02-17T05:11:01.825401+0000', 'last_peered': '2026-02-17T05:11:01.825401+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:01.825401+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:07.249 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:07.515 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:11:07.513+0000 7f6871c40640 0 --2- 10.20.193.100:0/4027127758 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7f686c05c490 0x7f686c137150 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=17771450592497570477 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:11:07.667 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:07.667 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1034,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:07.678 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:07.678 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:07.852 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:07.853 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:07.861 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":541,"stamp":"2026-02-17T05:11:06.391379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001676"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:10.863 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:11.041 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:11.042 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:11.051 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":543,"stamp":"2026-02-17T05:11:10.391815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001578"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:11.052 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:11.225 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:11.225 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:11.234 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":543,"stamp":"2026-02-17T05:11:10.391815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001578"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:11.235 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:11.235 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80099, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:01.825401+0000', 'last_change': '2026-02-17T05:11:01.825401+0000', 'last_active': '2026-02-17T05:11:01.825401+0000', 'last_peered': '2026-02-17T05:11:01.825401+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:01.825401+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:11.236 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:11.455 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:11.456 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1038,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:11.465 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:11.465 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:11.640 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:11.640 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:11.648 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":543,"stamp":"2026-02-17T05:11:10.391815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001578"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:14.650 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:14.826 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:14.826 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:14.835 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":545,"stamp":"2026-02-17T05:11:14.392307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001498"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084449,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:14.836 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:15.011 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:15.012 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:15.020 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":545,"stamp":"2026-02-17T05:11:14.392307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001498"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084449,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:15.022 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:15.022 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80099, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:01.825401+0000', 'last_change': '2026-02-17T05:11:01.825401+0000', 'last_active': '2026-02-17T05:11:01.825401+0000', 'last_peered': '2026-02-17T05:11:01.825401+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:01.825401+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:15.023 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:15.243 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:15.243 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1042,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:15.252 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:15.253 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:15.429 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:15.430 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:15.438 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":545,"stamp":"2026-02-17T05:11:14.392307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001498"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084449,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:18.440 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:18.619 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:18.620 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:18.629 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":547,"stamp":"2026-02-17T05:11:18.392852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001472"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084449,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:18.629 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:18.807 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:18.807 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:18.816 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":547,"stamp":"2026-02-17T05:11:18.392852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001472"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084449,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:18.816 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:18.816 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80099, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:01.825401+0000', 'last_change': '2026-02-17T05:11:01.825401+0000', 'last_active': '2026-02-17T05:11:01.825401+0000', 'last_peered': '2026-02-17T05:11:01.825401+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:01.825401+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:18.816 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:19.035 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:19.035 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1045,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:19.046 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:19.046 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:19.221 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:19.222 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:19.231 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":547,"stamp":"2026-02-17T05:11:18.392852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001472"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084449,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:22.232 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:22.411 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:22.412 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:22.421 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":549,"stamp":"2026-02-17T05:11:22.393309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084450,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:22.422 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:22.602 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:22.602 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:22.611 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":549,"stamp":"2026-02-17T05:11:22.393309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084450,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:22.612 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:22.612 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80099, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:01.825401+0000', 'last_change': '2026-02-17T05:11:01.825401+0000', 'last_active': '2026-02-17T05:11:01.825401+0000', 'last_peered': '2026-02-17T05:11:01.825401+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:01.825401+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:22.612 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:22.831 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:22.831 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1049,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:22.839 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:22.840 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:23.020 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:23.020 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:23.029 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":549,"stamp":"2026-02-17T05:11:22.393309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084450,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:26.032 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:26.214 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:26.215 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:26.223 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":550,"stamp":"2026-02-17T05:11:24.393550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001475"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084451,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:26.223 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:26.397 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:26.397 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:26.407 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2026-02-17T05:11:26.393780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001473"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084451,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:26.408 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:26.408 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80099, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:01.825401+0000', 'last_change': '2026-02-17T05:11:01.825401+0000', 'last_active': '2026-02-17T05:11:01.825401+0000', 'last_peered': '2026-02-17T05:11:01.825401+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:01.825401+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:26.408 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:26.627 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:26.627 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1053,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":16,"modified":"2026-02-17T05:09:58.382404+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:26.635 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:26.636 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:26.813 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:26.813 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:26.823 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2026-02-17T05:11:26.393780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001473"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084451,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:29.824 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:30.003 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:30.004 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:30.013 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":552,"stamp":"2026-02-17T05:11:28.394022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001436"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084451,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:30.014 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:30.191 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:30.191 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:30.200 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":552,"stamp":"2026-02-17T05:11:28.394022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001436"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084451,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:30.201 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:30.201 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80099, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:01.825401+0000', 'last_change': '2026-02-17T05:11:01.825401+0000', 'last_active': '2026-02-17T05:11:01.825401+0000', 'last_peered': '2026-02-17T05:11:01.825401+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:01.825401+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:30.202 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:30.418 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:30.418 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1057,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:30.427 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:30.427 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:30.602 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:30.603 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:30.611 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":553,"stamp":"2026-02-17T05:11:30.394308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001456"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084452,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:33.613 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:33.789 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:33.789 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:33.798 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2026-02-17T05:11:32.394545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001499"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084452,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:33.799 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:33.974 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:33.975 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:33.983 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2026-02-17T05:11:32.394545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001499"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80099,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:01.825401+0000","last_change":"2026-02-17T05:11:01.825401+0000","last_active":"2026-02-17T05:11:01.825401+0000","last_peered":"2026-02-17T05:11:01.825401+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:01.825401+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084452,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:33.984 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:33.984 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80099, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:01.825401+0000', 'last_change': '2026-02-17T05:11:01.825401+0000', 'last_active': '2026-02-17T05:11:01.825401+0000', 'last_peered': '2026-02-17T05:11:01.825401+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:01.825401+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:33.985 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:34.215 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:34.216 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1061,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:34.226 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:34.226 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:34.403 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:34.404 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:34.412 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":555,"stamp":"2026-02-17T05:11:34.394740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001430"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084453,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:37.414 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:37.593 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:37.593 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:37.602 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":556,"stamp":"2026-02-17T05:11:36.395000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001450"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084453,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:37.603 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:37.779 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:37.780 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:37.789 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":556,"stamp":"2026-02-17T05:11:36.395000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001450"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084453,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:37.790 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:37.790 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80103, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:31.825948+0000', 'last_change': '2026-02-17T05:11:31.825948+0000', 'last_active': '2026-02-17T05:11:31.825948+0000', 'last_peered': '2026-02-17T05:11:31.825948+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:31.825948+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:37.791 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:38.013 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:38.013 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1064,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:38.023 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:38.024 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:38.198 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:38.198 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:38.207 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":556,"stamp":"2026-02-17T05:11:36.395000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001450"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084453,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:41.208 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:41.385 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:41.386 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:41.395 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":558,"stamp":"2026-02-17T05:11:40.395576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001554"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084454,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:41.396 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:41.574 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:41.575 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:41.584 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":558,"stamp":"2026-02-17T05:11:40.395576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001554"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084454,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:41.585 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:41.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80103, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:31.825948+0000', 'last_change': '2026-02-17T05:11:31.825948+0000', 'last_active': '2026-02-17T05:11:31.825948+0000', 'last_peered': '2026-02-17T05:11:31.825948+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:31.825948+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:41.586 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:41.803 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:41.803 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1068,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:41.812 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:41.813 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:41.988 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:41.989 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:41.997 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":558,"stamp":"2026-02-17T05:11:40.395576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001554"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084454,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:44.999 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:45.175 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:45.176 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:45.184 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":560,"stamp":"2026-02-17T05:11:44.396107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001562"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084455,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:45.185 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:45.363 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:45.363 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:45.372 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":560,"stamp":"2026-02-17T05:11:44.396107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001562"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084455,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:45.373 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:45.373 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80103, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:31.825948+0000', 'last_change': '2026-02-17T05:11:31.825948+0000', 'last_active': '2026-02-17T05:11:31.825948+0000', 'last_peered': '2026-02-17T05:11:31.825948+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:31.825948+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:45.373 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:45.592 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:45.592 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1072,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:45.601 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:45.602 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:45.778 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:45.779 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:45.786 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":560,"stamp":"2026-02-17T05:11:44.396107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001562"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084455,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:48.788 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:48.963 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:48.964 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:48.971 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":562,"stamp":"2026-02-17T05:11:48.396674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001673"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084455,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:48.972 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:49.147 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:49.147 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:49.156 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":562,"stamp":"2026-02-17T05:11:48.396674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001673"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084455,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:49.157 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:49.157 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80103, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:31.825948+0000', 'last_change': '2026-02-17T05:11:31.825948+0000', 'last_active': '2026-02-17T05:11:31.825948+0000', 'last_peered': '2026-02-17T05:11:31.825948+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:31.825948+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:49.157 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:49.376 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:49.376 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1076,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:49.385 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:49.385 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:49.558 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:49.558 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:49.567 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":562,"stamp":"2026-02-17T05:11:48.396674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001673"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084455,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:52.568 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:52.748 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:52.748 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:52.757 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2026-02-17T05:11:52.397129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001552"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084456,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:52.757 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:52.935 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:52.936 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:52.945 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2026-02-17T05:11:52.397129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001552"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084456,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:52.946 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:52.947 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80103, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:31.825948+0000', 'last_change': '2026-02-17T05:11:31.825948+0000', 'last_active': '2026-02-17T05:11:31.825948+0000', 'last_peered': '2026-02-17T05:11:31.825948+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:31.825948+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:52.947 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:53.164 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:53.164 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1080,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:53.173 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:53.173 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:53.348 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:53.349 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:53.358 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2026-02-17T05:11:52.397129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001552"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084456,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:56.360 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:56.537 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:56.537 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:56.546 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":566,"stamp":"2026-02-17T05:11:56.397605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001497"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084457,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:56.547 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:56.722 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:56.723 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:56.732 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":566,"stamp":"2026-02-17T05:11:56.397605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001497"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084457,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:11:56.733 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:11:56.733 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80103, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:31.825948+0000', 'last_change': '2026-02-17T05:11:31.825948+0000', 'last_active': '2026-02-17T05:11:31.825948+0000', 'last_peered': '2026-02-17T05:11:31.825948+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:31.825948+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:11:56.733 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:11:57.001 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:11:56.999+0000 7fed64bc7640 0 --2- 10.20.193.100:0/2399504501 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7fed6005bb80 0x7fed6005bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:11:57.152 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:57.153 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1084,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:11:57.161 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:11:57.162 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:11:57.339 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:11:57.339 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:11:57.348 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":566,"stamp":"2026-02-17T05:11:56.397605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001497"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:07:58.490587+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:07:58.490587+0000","last_peered":"2026-02-17T05:07:58.490587+0000","last_clean":"2026-02-17T05:07:58.490587+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:07:58.490587+0000","last_undegraded":"2026-02-17T05:07:58.490587+0000","last_fullsized":"2026-02-17T05:07:58.490587+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084457,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:00.350 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:00.526 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:00.526 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:00.535 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":568,"stamp":"2026-02-17T05:12:00.398099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001424"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084458,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:00.536 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:00.714 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:00.714 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:00.722 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":568,"stamp":"2026-02-17T05:12:00.398099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001424"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084458,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:00.722 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:00.722 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80103, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:11:31.825948+0000', 'last_change': '2026-02-17T05:11:31.825948+0000', 'last_active': '2026-02-17T05:11:31.825948+0000', 'last_peered': '2026-02-17T05:11:31.825948+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:11:31.825948+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:00.722 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:00.940 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:00.941 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1087,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:00.949 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:00.950 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:01.219 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:01.220 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:01.232 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":568,"stamp":"2026-02-17T05:12:00.398099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001424"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80103,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:11:31.825948+0000","last_change":"2026-02-17T05:11:31.825948+0000","last_active":"2026-02-17T05:11:31.825948+0000","last_peered":"2026-02-17T05:11:31.825948+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:11:31.825948+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084458,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:04.235 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:04.416 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:04.417 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:04.425 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":570,"stamp":"2026-02-17T05:12:04.398616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001486"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80107,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826391+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826391+0000","last_peered":"2026-02-17T05:12:01.826391+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826391+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084459,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:04.426 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:04.607 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:04.607 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:04.616 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":570,"stamp":"2026-02-17T05:12:04.398616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001486"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80107,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826391+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826391+0000","last_peered":"2026-02-17T05:12:01.826391+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826391+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084459,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:04.617 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:04.617 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80107, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:01.826391+0000', 'last_change': '2026-02-17T05:12:01.826391+0000', 'last_active': '2026-02-17T05:12:01.826391+0000', 'last_peered': '2026-02-17T05:12:01.826391+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:01.826391+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:04.618 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:04.836 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:04.836 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1091,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:04.845 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:04.845 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:05.025 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:05.026 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:05.035 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":570,"stamp":"2026-02-17T05:12:04.398616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001486"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80107,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826391+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826391+0000","last_peered":"2026-02-17T05:12:01.826391+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826391+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084459,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:08.036 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:08.214 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:08.214 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:08.224 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":571,"stamp":"2026-02-17T05:12:06.398901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001544"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80107,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826391+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826391+0000","last_peered":"2026-02-17T05:12:01.826391+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826391+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084459,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:08.224 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:08.399 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:08.400 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:08.408 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":571,"stamp":"2026-02-17T05:12:06.398901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001544"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80107,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826391+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826391+0000","last_peered":"2026-02-17T05:12:01.826391+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826391+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084459,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:08.409 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:08.410 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80107, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:01.826391+0000', 'last_change': '2026-02-17T05:12:01.826391+0000', 'last_active': '2026-02-17T05:12:01.826391+0000', 'last_peered': '2026-02-17T05:12:01.826391+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:01.826391+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:08.410 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:08.627 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:08.627 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1095,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:08.637 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:08.637 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:08.816 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:08.817 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:08.826 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":572,"stamp":"2026-02-17T05:12:08.399175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001569"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80107,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826391+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826391+0000","last_peered":"2026-02-17T05:12:01.826391+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826391+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084459,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:11.828 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:12.005 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:12.006 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:12.014 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":573,"stamp":"2026-02-17T05:12:10.399379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001559"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80107,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826391+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826391+0000","last_peered":"2026-02-17T05:12:01.826391+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826391+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084460,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:12.015 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:12.193 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:12.193 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:12.202 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":573,"stamp":"2026-02-17T05:12:10.399379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001559"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80107,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826391+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826391+0000","last_peered":"2026-02-17T05:12:01.826391+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826391+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084460,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:12.203 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:12.203 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80107, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:01.826391+0000', 'last_change': '2026-02-17T05:12:01.826391+0000', 'last_active': '2026-02-17T05:12:01.826391+0000', 'last_peered': '2026-02-17T05:12:01.826391+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:01.826391+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:12.203 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:12.431 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:12.431 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1099,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:12.442 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:12.442 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:12.622 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:12.623 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:12.631 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":574,"stamp":"2026-02-17T05:12:12.399666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001567"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80107,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826391+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826391+0000","last_peered":"2026-02-17T05:12:01.826391+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826391+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084460,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:15.633 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:15.808 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:15.808 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:15.817 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":575,"stamp":"2026-02-17T05:12:14.399920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001600"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084461,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:15.817 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:15.993 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:15.993 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:16.001 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":575,"stamp":"2026-02-17T05:12:14.399920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001600"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084461,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:16.002 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:16.002 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80108, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:01.826417+0000', 'last_change': '2026-02-17T05:12:01.826391+0000', 'last_active': '2026-02-17T05:12:01.826417+0000', 'last_peered': '2026-02-17T05:12:01.826417+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:01.826417+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:16.003 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:16.222 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:16.222 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1103,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:16.233 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:16.233 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:16.409 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:16.409 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:16.417 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":576,"stamp":"2026-02-17T05:12:16.400167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084461,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:19.419 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:19.597 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:19.598 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:19.606 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":577,"stamp":"2026-02-17T05:12:18.400432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084461,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:19.607 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:19.789 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:19.789 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:19.797 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":577,"stamp":"2026-02-17T05:12:18.400432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084461,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:19.798 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:19.798 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80108, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:01.826417+0000', 'last_change': '2026-02-17T05:12:01.826391+0000', 'last_active': '2026-02-17T05:12:01.826417+0000', 'last_peered': '2026-02-17T05:12:01.826417+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:01.826417+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:19.799 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:20.067 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:12:20.066+0000 7f0de77fe640 0 --2- 10.20.193.100:0/4143015595 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7f0de807b8b0 0x7f0de807bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:12:20.220 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:20.220 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1107,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:20.229 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:20.229 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:20.407 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:20.408 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:20.416 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":578,"stamp":"2026-02-17T05:12:20.400666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001490"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084464,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:23.418 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:23.597 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:23.597 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:23.605 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":579,"stamp":"2026-02-17T05:12:22.400931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084464,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:23.606 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:23.780 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:23.780 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:23.789 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":579,"stamp":"2026-02-17T05:12:22.400931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084464,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:23.789 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:23.789 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80108, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:01.826417+0000', 'last_change': '2026-02-17T05:12:01.826391+0000', 'last_active': '2026-02-17T05:12:01.826417+0000', 'last_peered': '2026-02-17T05:12:01.826417+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:01.826417+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:23.790 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:24.008 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:24.008 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1110,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:24.017 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:24.017 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:24.193 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:24.194 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:24.203 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":579,"stamp":"2026-02-17T05:12:22.400931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084464,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:27.205 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:27.384 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:27.384 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:27.393 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":581,"stamp":"2026-02-17T05:12:26.401608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084465,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:27.393 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:27.567 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:27.568 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:27.576 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":581,"stamp":"2026-02-17T05:12:26.401608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084465,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:27.577 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:27.577 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80108, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:01.826417+0000', 'last_change': '2026-02-17T05:12:01.826391+0000', 'last_active': '2026-02-17T05:12:01.826417+0000', 'last_peered': '2026-02-17T05:12:01.826417+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:01.826417+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:27.577 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:27.796 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:27.796 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1114,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:27.805 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:27.805 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:27.980 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:27.980 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:27.988 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":581,"stamp":"2026-02-17T05:12:26.401608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084465,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:30.990 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:31.167 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:31.168 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:31.177 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":583,"stamp":"2026-02-17T05:12:30.402243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001810"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084466,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:31.178 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:31.351 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:31.352 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:31.359 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":583,"stamp":"2026-02-17T05:12:30.402243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001810"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084466,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:31.360 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:31.361 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80108, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:01.826417+0000', 'last_change': '2026-02-17T05:12:01.826391+0000', 'last_active': '2026-02-17T05:12:01.826417+0000', 'last_peered': '2026-02-17T05:12:01.826417+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:01.826417+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:31.361 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:31.577 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:31.577 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1118,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:31.586 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:31.586 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:31.763 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:31.764 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:31.772 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":583,"stamp":"2026-02-17T05:12:30.402243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001810"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80108,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:01.826417+0000","last_change":"2026-02-17T05:12:01.826391+0000","last_active":"2026-02-17T05:12:01.826417+0000","last_peered":"2026-02-17T05:12:01.826417+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:01.826417+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084466,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:34.774 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:34.951 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:34.952 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:34.960 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":585,"stamp":"2026-02-17T05:12:34.402857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001925"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084466,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:34.961 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:35.140 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:35.141 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:35.149 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":585,"stamp":"2026-02-17T05:12:34.402857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001925"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084466,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:35.150 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:35.150 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80112, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:31.826945+0000', 'last_change': '2026-02-17T05:12:31.826945+0000', 'last_active': '2026-02-17T05:12:31.826945+0000', 'last_peered': '2026-02-17T05:12:31.826945+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:31.826945+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:35.151 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:35.368 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:35.369 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1122,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:35.378 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:35.378 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:35.553 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:35.553 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:35.561 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":585,"stamp":"2026-02-17T05:12:34.402857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001925"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084466,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:38.563 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:38.738 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:38.738 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:38.747 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":587,"stamp":"2026-02-17T05:12:38.403500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001892"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084467,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:38.747 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:38.925 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:38.925 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:38.934 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":587,"stamp":"2026-02-17T05:12:38.403500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001892"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084467,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:38.934 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:38.934 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80112, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:31.826945+0000', 'last_change': '2026-02-17T05:12:31.826945+0000', 'last_active': '2026-02-17T05:12:31.826945+0000', 'last_peered': '2026-02-17T05:12:31.826945+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:31.826945+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:38.935 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:39.153 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:39.153 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1126,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:39.162 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:39.163 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:39.341 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:39.341 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:39.350 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":587,"stamp":"2026-02-17T05:12:38.403500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001892"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084467,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:42.352 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:42.528 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:42.529 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:42.539 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2026-02-17T05:12:42.404010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001767"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084468,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:42.540 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:42.714 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:42.715 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:42.723 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2026-02-17T05:12:42.404010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001767"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084468,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:42.724 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:42.724 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80112, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:31.826945+0000', 'last_change': '2026-02-17T05:12:31.826945+0000', 'last_active': '2026-02-17T05:12:31.826945+0000', 'last_peered': '2026-02-17T05:12:31.826945+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:31.826945+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:42.725 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:42.943 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:42.943 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1129,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:42.953 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:42.953 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:43.129 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:43.129 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:43.137 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2026-02-17T05:12:42.404010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001767"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084468,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:46.139 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:46.317 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:46.317 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:46.326 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":590,"stamp":"2026-02-17T05:12:44.404251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001723"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084468,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:46.327 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:46.502 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:46.502 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:46.511 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":591,"stamp":"2026-02-17T05:12:46.404560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001703"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084469,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:46.512 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:46.512 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80112, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:31.826945+0000', 'last_change': '2026-02-17T05:12:31.826945+0000', 'last_active': '2026-02-17T05:12:31.826945+0000', 'last_peered': '2026-02-17T05:12:31.826945+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:31.826945+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:46.513 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:46.728 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:46.729 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1133,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:46.737 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:46.737 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:46.914 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:46.914 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:46.923 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":591,"stamp":"2026-02-17T05:12:46.404560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001703"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084469,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:49.924 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:50.100 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:50.101 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:50.109 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":592,"stamp":"2026-02-17T05:12:48.404798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001629"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084469,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:50.110 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:50.284 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:50.285 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:50.293 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":592,"stamp":"2026-02-17T05:12:48.404798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001629"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084469,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:50.294 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:50.294 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80112, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:31.826945+0000', 'last_change': '2026-02-17T05:12:31.826945+0000', 'last_active': '2026-02-17T05:12:31.826945+0000', 'last_peered': '2026-02-17T05:12:31.826945+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:31.826945+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:50.295 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:50.512 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:50.512 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1137,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:50.522 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:50.522 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:50.695 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:50.696 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:50.704 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":593,"stamp":"2026-02-17T05:12:50.405021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001521"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084470,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:53.706 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:53.885 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:53.885 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:53.894 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":594,"stamp":"2026-02-17T05:12:52.405233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001481"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084470,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:53.895 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:54.072 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:54.072 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:54.081 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":594,"stamp":"2026-02-17T05:12:52.405233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001481"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084470,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:54.082 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:54.082 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80112, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:31.826945+0000', 'last_change': '2026-02-17T05:12:31.826945+0000', 'last_active': '2026-02-17T05:12:31.826945+0000', 'last_peered': '2026-02-17T05:12:31.826945+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:31.826945+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:54.082 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:54.300 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:54.300 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1141,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:54.310 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:54.310 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:54.489 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:54.490 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:54.499 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":595,"stamp":"2026-02-17T05:12:54.405532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001522"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084470,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:57.501 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:57.678 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:57.678 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:57.687 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2026-02-17T05:12:56.405781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084471,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:57.688 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:57.864 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:57.865 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:57.873 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2026-02-17T05:12:56.405781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084471,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:12:57.874 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:12:57.874 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80112, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:31.826945+0000', 'last_change': '2026-02-17T05:12:31.826945+0000', 'last_active': '2026-02-17T05:12:31.826945+0000', 'last_peered': '2026-02-17T05:12:31.826945+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:31.826945+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:12:57.875 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:12:58.094 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:58.094 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1145,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:12:58.103 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:12:58.103 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:12:58.279 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:12:58.280 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:12:58.289 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2026-02-17T05:12:56.405781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084471,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:01.291 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:01.563 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:13:01.562+0000 7f7fb4f52640 0 --2- 10.20.193.100:0/3289194672 >> [v2:10.20.193.100:3300/0,v1:10.20.193.100:6789/0] conn(0x7f7fb005bb80 0x7f7fb012f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:13:01.670 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:01.670 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:01.678 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2026-02-17T05:13:00.406381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001582"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084472,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:01.679 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:01.853 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:01.854 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:01.862 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2026-02-17T05:13:00.406381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001582"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084472,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:01.863 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:01.863 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80112, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:12:31.826945+0000', 'last_change': '2026-02-17T05:12:31.826945+0000', 'last_active': '2026-02-17T05:12:31.826945+0000', 'last_peered': '2026-02-17T05:12:31.826945+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:12:31.826945+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:01.863 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:02.084 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:02.084 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1149,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:02.094 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:02.094 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:02.270 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:02.271 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:02.279 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2026-02-17T05:13:00.406381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001582"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80112,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:12:31.826945+0000","last_change":"2026-02-17T05:12:31.826945+0000","last_active":"2026-02-17T05:12:31.826945+0000","last_peered":"2026-02-17T05:12:31.826945+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:12:31.826945+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084472,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:05.281 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:05.459 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:05.460 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:05.469 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2026-02-17T05:13:04.407018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001784"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084472,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:05.470 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:05.643 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:05.644 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:05.652 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2026-02-17T05:13:04.407018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001784"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084472,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:05.653 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:05.654 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80116, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:01.827436+0000', 'last_change': '2026-02-17T05:13:01.827436+0000', 'last_active': '2026-02-17T05:13:01.827436+0000', 'last_peered': '2026-02-17T05:13:01.827436+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:01.827436+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:05.654 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:05.871 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:05.871 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1152,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:05.880 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:05.881 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:06.057 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:06.057 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:06.065 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2026-02-17T05:13:04.407018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001784"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084472,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:09.067 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:09.250 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:09.250 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:09.255 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":602,"stamp":"2026-02-17T05:13:08.407525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001743"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084473,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:09.255 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:09.441 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:09.442 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:09.451 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":602,"stamp":"2026-02-17T05:13:08.407525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001743"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084473,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:09.452 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:09.452 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80116, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:01.827436+0000', 'last_change': '2026-02-17T05:13:01.827436+0000', 'last_active': '2026-02-17T05:13:01.827436+0000', 'last_peered': '2026-02-17T05:13:01.827436+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:01.827436+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:09.453 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:09.673 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:09.673 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1156,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:09.682 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:09.682 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:09.861 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:09.861 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:09.870 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":602,"stamp":"2026-02-17T05:13:08.407525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001743"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084473,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:12.872 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:13.048 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:13.049 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:13.056 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":604,"stamp":"2026-02-17T05:13:12.408142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001761"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084474,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:13.057 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:13.232 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:13.233 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:13.241 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":604,"stamp":"2026-02-17T05:13:12.408142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001761"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084474,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:13.242 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:13.242 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80116, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:01.827436+0000', 'last_change': '2026-02-17T05:13:01.827436+0000', 'last_active': '2026-02-17T05:13:01.827436+0000', 'last_peered': '2026-02-17T05:13:01.827436+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:01.827436+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:13.243 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:13.461 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:13.461 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1160,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:13.471 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:13.471 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:13.649 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:13.649 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:13.658 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":604,"stamp":"2026-02-17T05:13:12.408142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001761"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084474,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:16.660 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:16.834 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:16.834 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:16.842 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":606,"stamp":"2026-02-17T05:13:16.408692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001674"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084475,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:16.843 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:17.021 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:17.021 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:17.029 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":606,"stamp":"2026-02-17T05:13:16.408692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001674"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084475,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:17.030 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:17.030 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80116, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:01.827436+0000', 'last_change': '2026-02-17T05:13:01.827436+0000', 'last_active': '2026-02-17T05:13:01.827436+0000', 'last_peered': '2026-02-17T05:13:01.827436+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:01.827436+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:17.031 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:17.245 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:17.246 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1164,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:17.255 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:17.255 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:17.431 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:17.432 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:17.440 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":606,"stamp":"2026-02-17T05:13:16.408692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001674"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084475,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:20.443 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:20.620 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:20.620 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:20.629 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":608,"stamp":"2026-02-17T05:13:20.409147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:20.630 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:20.808 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:20.808 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:20.817 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":608,"stamp":"2026-02-17T05:13:20.409147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:20.818 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:20.818 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80116, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:01.827436+0000', 'last_change': '2026-02-17T05:13:01.827436+0000', 'last_active': '2026-02-17T05:13:01.827436+0000', 'last_peered': '2026-02-17T05:13:01.827436+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:01.827436+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:20.819 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:21.039 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:21.040 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1167,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:21.048 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:21.049 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:21.226 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:21.226 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:21.235 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":608,"stamp":"2026-02-17T05:13:20.409147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:24.237 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:24.415 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:24.415 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:24.424 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":610,"stamp":"2026-02-17T05:13:24.409742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:24.424 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:24.605 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:24.606 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:24.616 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":610,"stamp":"2026-02-17T05:13:24.409742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:24.616 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:24.616 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80116, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:01.827436+0000', 'last_change': '2026-02-17T05:13:01.827436+0000', 'last_active': '2026-02-17T05:13:01.827436+0000', 'last_peered': '2026-02-17T05:13:01.827436+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:01.827436+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:24.616 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:24.840 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:24.841 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1171,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:24.849 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:24.849 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:25.020 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:25.021 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:25.029 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":610,"stamp":"2026-02-17T05:13:24.409742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001599"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:28.030 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:28.209 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:28.210 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:28.218 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":611,"stamp":"2026-02-17T05:13:26.410042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084478,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:28.220 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:28.394 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:28.394 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:28.402 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":611,"stamp":"2026-02-17T05:13:26.410042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084478,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:28.403 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:28.404 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80116, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:01.827436+0000', 'last_change': '2026-02-17T05:13:01.827436+0000', 'last_active': '2026-02-17T05:13:01.827436+0000', 'last_peered': '2026-02-17T05:13:01.827436+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:01.827436+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:28.404 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:28.622 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:28.622 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1175,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:28.632 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:28.632 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:28.807 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:28.807 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:28.815 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":612,"stamp":"2026-02-17T05:13:28.410363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001670"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084478,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:31.818 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:31.997 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:31.998 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:32.006 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":613,"stamp":"2026-02-17T05:13:30.410629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:32.007 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:32.184 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:32.184 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:32.194 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":613,"stamp":"2026-02-17T05:13:30.410629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:32.195 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:32.195 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80116, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:01.827436+0000', 'last_change': '2026-02-17T05:13:01.827436+0000', 'last_active': '2026-02-17T05:13:01.827436+0000', 'last_peered': '2026-02-17T05:13:01.827436+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:01.827436+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:32.196 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:32.412 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:32.413 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1179,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:32.421 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:32.422 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:32.599 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:32.600 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:32.610 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":614,"stamp":"2026-02-17T05:13:32.410915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001768"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80116,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:01.827436+0000","last_change":"2026-02-17T05:13:01.827436+0000","last_active":"2026-02-17T05:13:01.827436+0000","last_peered":"2026-02-17T05:13:01.827436+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:01.827436+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:35.612 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:35.791 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:35.792 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:35.801 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":615,"stamp":"2026-02-17T05:13:34.411177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001747"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:35.802 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:35.979 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:35.980 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:35.988 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":615,"stamp":"2026-02-17T05:13:34.411177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001747"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:35.989 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:35.989 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80120, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:31.827893+0000', 'last_change': '2026-02-17T05:13:31.827893+0000', 'last_active': '2026-02-17T05:13:31.827893+0000', 'last_peered': '2026-02-17T05:13:31.827893+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:31.827893+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:35.990 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:36.208 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:36.208 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1183,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:36.218 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:36.218 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:36.392 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:36.392 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:36.400 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":615,"stamp":"2026-02-17T05:13:34.411177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001747"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:39.402 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:39.578 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:39.579 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:39.588 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":617,"stamp":"2026-02-17T05:13:38.411774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001732"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084480,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:39.589 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:39.764 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:39.765 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:39.773 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":617,"stamp":"2026-02-17T05:13:38.411774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001732"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084480,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:39.774 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:39.774 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80120, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:31.827893+0000', 'last_change': '2026-02-17T05:13:31.827893+0000', 'last_active': '2026-02-17T05:13:31.827893+0000', 'last_peered': '2026-02-17T05:13:31.827893+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:31.827893+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:39.774 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:40.001 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:40.002 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1186,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:40.011 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:40.011 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:40.187 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:40.187 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:40.196 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":617,"stamp":"2026-02-17T05:13:38.411774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001732"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084480,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:43.198 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:43.376 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:43.376 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:43.385 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":619,"stamp":"2026-02-17T05:13:42.412444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001815"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084481,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:43.385 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:43.560 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:43.560 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:43.570 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":619,"stamp":"2026-02-17T05:13:42.412444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001815"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084481,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:43.570 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:43.570 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80120, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:31.827893+0000', 'last_change': '2026-02-17T05:13:31.827893+0000', 'last_active': '2026-02-17T05:13:31.827893+0000', 'last_peered': '2026-02-17T05:13:31.827893+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:31.827893+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:43.571 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:43.790 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:43.790 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1190,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:43.799 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:43.799 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:43.970 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:43.971 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:43.978 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":619,"stamp":"2026-02-17T05:13:42.412444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001815"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084481,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:46.979 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:47.156 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:47.157 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:47.165 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":621,"stamp":"2026-02-17T05:13:46.412865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001687"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084482,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:47.167 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:47.346 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:47.347 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:47.355 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":621,"stamp":"2026-02-17T05:13:46.412865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001687"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084482,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:47.355 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:47.356 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80120, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:31.827893+0000', 'last_change': '2026-02-17T05:13:31.827893+0000', 'last_active': '2026-02-17T05:13:31.827893+0000', 'last_peered': '2026-02-17T05:13:31.827893+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:31.827893+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:47.356 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:47.572 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:47.572 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1194,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:47.582 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:47.582 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:47.759 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:47.760 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:47.768 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":621,"stamp":"2026-02-17T05:13:46.412865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001687"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084482,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:50.770 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:50.947 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:50.947 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:50.956 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":623,"stamp":"2026-02-17T05:13:50.413434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084483,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:50.956 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:51.131 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:51.132 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:51.140 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":623,"stamp":"2026-02-17T05:13:50.413434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084483,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:51.140 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:51.140 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80120, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:31.827893+0000', 'last_change': '2026-02-17T05:13:31.827893+0000', 'last_active': '2026-02-17T05:13:31.827893+0000', 'last_peered': '2026-02-17T05:13:31.827893+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:31.827893+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:51.141 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:51.359 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:51.359 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1198,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:51.368 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:51.368 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:51.545 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:51.546 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:51.555 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":623,"stamp":"2026-02-17T05:13:50.413434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084483,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:54.556 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:54.733 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:54.734 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:54.741 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":625,"stamp":"2026-02-17T05:13:54.413952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001508"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084483,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:54.743 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:54.918 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:54.919 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:54.928 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":625,"stamp":"2026-02-17T05:13:54.413952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001508"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084483,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:54.929 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:54.929 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80120, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:31.827893+0000', 'last_change': '2026-02-17T05:13:31.827893+0000', 'last_active': '2026-02-17T05:13:31.827893+0000', 'last_peered': '2026-02-17T05:13:31.827893+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:31.827893+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:54.929 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:55.145 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:55.145 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1202,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:55.154 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:55.154 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:55.332 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:55.333 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:55.341 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":625,"stamp":"2026-02-17T05:13:54.413952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001508"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084483,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:58.344 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:58.525 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:58.525 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:58.533 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2026-02-17T05:13:58.414511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001646"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084484,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:58.535 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:58.714 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:58.714 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:58.723 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2026-02-17T05:13:58.414511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001646"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084484,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:13:58.724 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:13:58.724 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80120, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:31.827893+0000', 'last_change': '2026-02-17T05:13:31.827893+0000', 'last_active': '2026-02-17T05:13:31.827893+0000', 'last_peered': '2026-02-17T05:13:31.827893+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:31.827893+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:13:58.725 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:13:58.944 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:58.944 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1205,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:13:58.954 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:13:58.954 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:13:59.128 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:13:59.129 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:13:59.137 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2026-02-17T05:13:58.414511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001646"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:09:58.499494+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:09:58.499494+0000","last_peered":"2026-02-17T05:09:58.499494+0000","last_clean":"2026-02-17T05:09:58.499494+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:09:58.499494+0000","last_undegraded":"2026-02-17T05:09:58.499494+0000","last_fullsized":"2026-02-17T05:09:58.499494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084484,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:02.138 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:02.317 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:02.317 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:02.326 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":628,"stamp":"2026-02-17T05:14:00.414714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084485,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:02.327 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:02.504 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:02.504 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:02.513 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":629,"stamp":"2026-02-17T05:14:02.415012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001577"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084485,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:02.514 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:02.514 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80120, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:13:31.827893+0000', 'last_change': '2026-02-17T05:13:31.827893+0000', 'last_active': '2026-02-17T05:13:31.827893+0000', 'last_peered': '2026-02-17T05:13:31.827893+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:13:31.827893+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:02.515 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:02.735 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:02.735 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1209,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:02.745 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:02.745 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:02.921 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:02.921 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:02.930 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":629,"stamp":"2026-02-17T05:14:02.415012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001577"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80120,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:13:31.827893+0000","last_change":"2026-02-17T05:13:31.827893+0000","last_active":"2026-02-17T05:13:31.827893+0000","last_peered":"2026-02-17T05:13:31.827893+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:13:31.827893+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084485,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:05.932 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:06.110 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:06.110 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:06.119 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":630,"stamp":"2026-02-17T05:14:04.415262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001568"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80124,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828299+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828299+0000","last_peered":"2026-02-17T05:14:01.828299+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828299+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084485,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:06.119 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:06.293 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:06.294 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:06.302 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":630,"stamp":"2026-02-17T05:14:04.415262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001568"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80124,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828299+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828299+0000","last_peered":"2026-02-17T05:14:01.828299+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828299+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084485,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:06.303 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:06.303 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80124, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:01.828299+0000', 'last_change': '2026-02-17T05:14:01.828299+0000', 'last_active': '2026-02-17T05:14:01.828299+0000', 'last_peered': '2026-02-17T05:14:01.828299+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:01.828299+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:06.304 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:06.524 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:06.524 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1213,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:06.534 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:06.534 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:06.710 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:06.710 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:06.718 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":631,"stamp":"2026-02-17T05:14:06.415575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001623"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80124,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828299+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828299+0000","last_peered":"2026-02-17T05:14:01.828299+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828299+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084486,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:09.720 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:09.898 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:09.898 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:09.907 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":632,"stamp":"2026-02-17T05:14:08.415833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80124,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828299+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828299+0000","last_peered":"2026-02-17T05:14:01.828299+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828299+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084486,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:09.908 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:10.085 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:10.085 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:10.094 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":632,"stamp":"2026-02-17T05:14:08.415833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80124,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828299+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828299+0000","last_peered":"2026-02-17T05:14:01.828299+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828299+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084486,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:10.094 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:10.094 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80124, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:01.828299+0000', 'last_change': '2026-02-17T05:14:01.828299+0000', 'last_active': '2026-02-17T05:14:01.828299+0000', 'last_peered': '2026-02-17T05:14:01.828299+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:01.828299+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:10.095 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:10.309 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:10.309 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1217,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:10.318 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:10.318 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:10.492 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:10.493 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:10.501 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":633,"stamp":"2026-02-17T05:14:10.416122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001610"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80124,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828299+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828299+0000","last_peered":"2026-02-17T05:14:01.828299+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828299+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084487,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:13.503 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:13.681 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:13.682 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:13.690 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":634,"stamp":"2026-02-17T05:14:12.416387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001673"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80124,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828299+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828299+0000","last_peered":"2026-02-17T05:14:01.828299+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828299+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084487,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:13.691 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:13.872 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:13.873 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:13.881 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":634,"stamp":"2026-02-17T05:14:12.416387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001673"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80124,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828299+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828299+0000","last_peered":"2026-02-17T05:14:01.828299+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828299+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084487,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:13.882 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:13.882 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80124, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:01.828299+0000', 'last_change': '2026-02-17T05:14:01.828299+0000', 'last_active': '2026-02-17T05:14:01.828299+0000', 'last_peered': '2026-02-17T05:14:01.828299+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:01.828299+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:13.883 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:14.103 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:14.104 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1221,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:14.114 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:14.114 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:14.293 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:14.294 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:14.302 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":634,"stamp":"2026-02-17T05:14:12.416387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001673"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80124,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828299+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828299+0000","last_peered":"2026-02-17T05:14:01.828299+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828299+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084487,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:17.303 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:17.475 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:17.475 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:17.484 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2026-02-17T05:14:16.416846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001584"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084488,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:17.484 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:17.660 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:17.661 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:17.669 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2026-02-17T05:14:16.416846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001584"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084488,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:17.670 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:17.671 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80125, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:01.828335+0000', 'last_change': '2026-02-17T05:14:01.828299+0000', 'last_active': '2026-02-17T05:14:01.828335+0000', 'last_peered': '2026-02-17T05:14:01.828335+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:01.828335+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:17.671 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:17.889 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:17.889 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1224,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:17.898 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:17.898 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:18.079 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:18.079 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:18.088 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2026-02-17T05:14:16.416846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001584"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084488,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:21.090 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:21.268 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:21.268 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:21.277 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2026-02-17T05:14:20.417247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001413"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:21.278 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:21.454 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:21.455 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:21.463 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2026-02-17T05:14:20.417247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001413"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:21.464 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:21.465 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80125, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:01.828335+0000', 'last_change': '2026-02-17T05:14:01.828299+0000', 'last_active': '2026-02-17T05:14:01.828335+0000', 'last_peered': '2026-02-17T05:14:01.828335+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:01.828335+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:21.465 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:21.682 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:21.682 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1228,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:21.692 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:21.692 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:21.867 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:21.867 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:21.876 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2026-02-17T05:14:20.417247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001413"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:24.878 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:25.055 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:25.055 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:25.064 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":640,"stamp":"2026-02-17T05:14:24.417703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001315"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:25.065 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:25.240 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:25.241 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:25.249 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":640,"stamp":"2026-02-17T05:14:24.417703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001315"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:25.250 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:25.250 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80125, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:01.828335+0000', 'last_change': '2026-02-17T05:14:01.828299+0000', 'last_active': '2026-02-17T05:14:01.828335+0000', 'last_peered': '2026-02-17T05:14:01.828335+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:01.828335+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:25.251 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:25.469 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:25.469 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1232,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:25.479 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:25.479 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:25.654 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:25.654 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:25.662 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":640,"stamp":"2026-02-17T05:14:24.417703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001315"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:28.663 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:28.838 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:28.839 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:28.847 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":642,"stamp":"2026-02-17T05:14:28.418206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001359"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084490,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:28.849 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:29.024 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:29.024 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:29.033 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":642,"stamp":"2026-02-17T05:14:28.418206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001359"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084490,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:29.034 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:29.034 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80125, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:01.828335+0000', 'last_change': '2026-02-17T05:14:01.828299+0000', 'last_active': '2026-02-17T05:14:01.828335+0000', 'last_peered': '2026-02-17T05:14:01.828335+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:01.828335+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:29.034 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:29.251 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:29.252 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1236,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:29.260 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:29.261 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:29.435 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:29.436 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:29.444 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":642,"stamp":"2026-02-17T05:14:28.418206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001359"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084490,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:32.446 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:32.624 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:32.625 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:32.633 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":644,"stamp":"2026-02-17T05:14:32.418641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001393"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084491,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:32.634 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:32.811 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:32.812 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:32.820 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":644,"stamp":"2026-02-17T05:14:32.418641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001393"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084491,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:32.821 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:32.821 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80125, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:01.828335+0000', 'last_change': '2026-02-17T05:14:01.828299+0000', 'last_active': '2026-02-17T05:14:01.828335+0000', 'last_peered': '2026-02-17T05:14:01.828335+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:01.828335+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:32.821 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:33.040 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:33.040 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1239,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:33.049 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:33.050 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:33.222 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:33.223 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:33.231 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":644,"stamp":"2026-02-17T05:14:32.418641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001393"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80125,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:01.828335+0000","last_change":"2026-02-17T05:14:01.828299+0000","last_active":"2026-02-17T05:14:01.828335+0000","last_peered":"2026-02-17T05:14:01.828335+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:01.828335+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084491,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:36.233 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:36.409 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:36.410 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:36.418 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":645,"stamp":"2026-02-17T05:14:34.418835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001336"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084491,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:36.419 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:36.592 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:36.592 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:36.601 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":646,"stamp":"2026-02-17T05:14:36.419084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001380"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084492,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:36.601 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:36.602 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80129, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:31.828774+0000', 'last_change': '2026-02-17T05:14:31.828774+0000', 'last_active': '2026-02-17T05:14:31.828774+0000', 'last_peered': '2026-02-17T05:14:31.828774+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:31.828774+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:36.602 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:36.816 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:36.816 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1243,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:36.825 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:36.826 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:37.001 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:37.002 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:37.010 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":646,"stamp":"2026-02-17T05:14:36.419084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001380"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084492,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:40.011 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:40.192 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:40.193 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:40.201 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":647,"stamp":"2026-02-17T05:14:38.419347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001383"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084492,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:40.203 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:40.378 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:40.378 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:40.387 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":647,"stamp":"2026-02-17T05:14:38.419347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001383"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084492,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:40.387 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:40.387 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80129, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:31.828774+0000', 'last_change': '2026-02-17T05:14:31.828774+0000', 'last_active': '2026-02-17T05:14:31.828774+0000', 'last_peered': '2026-02-17T05:14:31.828774+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:31.828774+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:40.387 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:40.605 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:40.606 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1247,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:40.615 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:40.616 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:40.796 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:40.796 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:40.805 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":648,"stamp":"2026-02-17T05:14:40.419652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001446"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084493,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:43.806 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:43.984 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:43.985 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:43.993 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":649,"stamp":"2026-02-17T05:14:42.419907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001452"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084493,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:43.995 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:44.169 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:44.170 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:44.178 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":649,"stamp":"2026-02-17T05:14:42.419907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001452"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084493,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:44.179 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:44.179 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80129, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:31.828774+0000', 'last_change': '2026-02-17T05:14:31.828774+0000', 'last_active': '2026-02-17T05:14:31.828774+0000', 'last_peered': '2026-02-17T05:14:31.828774+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:31.828774+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:44.179 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:44.395 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:44.395 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1251,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:44.403 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:44.403 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:44.673 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:14:44.672+0000 7fd452642640 0 --2- 10.20.193.100:0/326494542 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7fd44c137860 0x7fd44c1307a0 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:14:44.780 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:44.781 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:44.790 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":650,"stamp":"2026-02-17T05:14:44.420202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084493,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:47.793 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:47.969 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:47.969 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:47.977 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":651,"stamp":"2026-02-17T05:14:46.420472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001637"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084494,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:47.978 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:48.154 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:48.155 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:48.163 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":651,"stamp":"2026-02-17T05:14:46.420472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001637"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084494,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:48.164 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:48.164 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80129, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:31.828774+0000', 'last_change': '2026-02-17T05:14:31.828774+0000', 'last_active': '2026-02-17T05:14:31.828774+0000', 'last_peered': '2026-02-17T05:14:31.828774+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:31.828774+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:48.165 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:48.434 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:14:48.433+0000 7f2855c81640 0 --2- 10.20.193.100:0/213690227 >> v2:10.20.193.100:3300/0 conn(0x7f285007e8b0 0x7f285007ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:14:48.603 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:48.603 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1255,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:48.613 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:48.614 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:48.886 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:14:48.885+0000 7f5ce4a40640 0 --2- 10.20.193.100:0/1940637510 >> [v2:10.20.193.100:3300/0,v1:10.20.193.100:6789/0] conn(0x7f5ce005c490 0x7f5ce010fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:14:48.995 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:48.995 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:49.004 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":652,"stamp":"2026-02-17T05:14:48.420722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001638"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084494,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:52.006 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:52.185 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:52.186 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:52.194 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":653,"stamp":"2026-02-17T05:14:50.420936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084495,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:52.195 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:52.374 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:52.375 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:52.383 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":653,"stamp":"2026-02-17T05:14:50.420936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084495,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:52.385 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:52.385 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80129, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:31.828774+0000', 'last_change': '2026-02-17T05:14:31.828774+0000', 'last_active': '2026-02-17T05:14:31.828774+0000', 'last_peered': '2026-02-17T05:14:31.828774+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:31.828774+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:52.386 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:52.606 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:52.606 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1259,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:52.615 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:52.615 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:52.791 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:52.792 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:52.800 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":654,"stamp":"2026-02-17T05:14:52.421262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001610"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084495,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:55.801 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:55.977 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:55.978 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:55.986 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2026-02-17T05:14:54.421492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001584"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084495,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:55.986 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:57.056 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:57.056 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:57.066 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":656,"stamp":"2026-02-17T05:14:56.421791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084496,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:14:57.067 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:14:57.067 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80129, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:31.828774+0000', 'last_change': '2026-02-17T05:14:31.828774+0000', 'last_active': '2026-02-17T05:14:31.828774+0000', 'last_peered': '2026-02-17T05:14:31.828774+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:31.828774+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:14:57.068 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:14:57.287 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:57.287 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1264,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:14:57.296 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:14:57.297 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:14:57.475 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:14:57.476 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:14:57.484 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":656,"stamp":"2026-02-17T05:14:56.421791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084496,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:00.487 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:00.663 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:00.663 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:00.671 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":658,"stamp":"2026-02-17T05:15:00.422334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001612"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:00.672 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:00.849 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:00.849 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:00.859 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":658,"stamp":"2026-02-17T05:15:00.422334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001612"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:00.860 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:00.860 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80129, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:14:31.828774+0000', 'last_change': '2026-02-17T05:14:31.828774+0000', 'last_active': '2026-02-17T05:14:31.828774+0000', 'last_peered': '2026-02-17T05:14:31.828774+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:14:31.828774+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:00.860 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:01.079 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:01.079 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1268,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:01.089 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:01.090 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:01.357 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:15:01.356+0000 7f88967fc640 0 --2- 10.20.193.100:0/1104105539 >> [v2:10.20.193.100:3302/0,v1:10.20.193.100:6791/0] conn(0x7f889807c1c0 0x7f8898175150 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:15:01.467 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:01.468 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:01.476 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":658,"stamp":"2026-02-17T05:15:00.422334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001612"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80129,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:14:31.828774+0000","last_change":"2026-02-17T05:14:31.828774+0000","last_active":"2026-02-17T05:14:31.828774+0000","last_peered":"2026-02-17T05:14:31.828774+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:14:31.828774+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:04.478 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:04.655 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:04.655 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:04.663 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":660,"stamp":"2026-02-17T05:15:04.422839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001576"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:04.664 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:04.838 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:04.839 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:04.847 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":660,"stamp":"2026-02-17T05:15:04.422839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001576"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:04.848 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:04.848 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80133, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:01.829462+0000', 'last_change': '2026-02-17T05:15:01.829462+0000', 'last_active': '2026-02-17T05:15:01.829462+0000', 'last_peered': '2026-02-17T05:15:01.829462+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:01.829462+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:04.848 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:05.066 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:05.066 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1272,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:05.076 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:05.077 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:05.250 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:05.250 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:05.258 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":660,"stamp":"2026-02-17T05:15:04.422839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001576"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:08.260 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:08.435 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:08.435 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:08.444 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":662,"stamp":"2026-02-17T05:15:08.423298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084498,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:08.444 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:08.618 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:08.618 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:08.627 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":662,"stamp":"2026-02-17T05:15:08.423298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084498,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:08.627 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:08.627 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80133, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:01.829462+0000', 'last_change': '2026-02-17T05:15:01.829462+0000', 'last_active': '2026-02-17T05:15:01.829462+0000', 'last_peered': '2026-02-17T05:15:01.829462+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:01.829462+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:08.627 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:08.846 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:08.846 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1275,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:08.855 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:08.855 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:09.028 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:09.029 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:09.037 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":662,"stamp":"2026-02-17T05:15:08.423298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084498,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:12.038 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:12.215 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:12.215 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:12.224 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2026-02-17T05:15:10.423551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:12.226 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:12.402 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:12.402 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:12.411 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2026-02-17T05:15:10.423551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:12.412 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:12.412 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80133, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:01.829462+0000', 'last_change': '2026-02-17T05:15:01.829462+0000', 'last_active': '2026-02-17T05:15:01.829462+0000', 'last_peered': '2026-02-17T05:15:01.829462+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:01.829462+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:12.412 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:12.636 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:12.637 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1279,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:12.646 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:12.646 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:12.827 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:12.827 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:12.836 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":664,"stamp":"2026-02-17T05:15:12.423860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:15.838 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:16.016 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:16.017 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:16.025 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":665,"stamp":"2026-02-17T05:15:14.424121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001546"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:16.026 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:16.197 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:16.197 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:16.206 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":665,"stamp":"2026-02-17T05:15:14.424121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001546"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:16.207 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:16.207 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80133, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:01.829462+0000', 'last_change': '2026-02-17T05:15:01.829462+0000', 'last_active': '2026-02-17T05:15:01.829462+0000', 'last_peered': '2026-02-17T05:15:01.829462+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:01.829462+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:16.208 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:16.428 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:16.428 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1283,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:16.438 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:16.438 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:16.612 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:16.612 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:16.621 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":666,"stamp":"2026-02-17T05:15:16.424451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001612"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084500,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:19.623 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:20.695 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:20.695 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:20.705 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2026-02-17T05:15:20.424993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001695"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084501,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:20.706 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:20.882 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:20.882 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:20.891 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2026-02-17T05:15:20.424993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001695"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084501,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:20.892 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:20.892 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80133, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:01.829462+0000', 'last_change': '2026-02-17T05:15:01.829462+0000', 'last_active': '2026-02-17T05:15:01.829462+0000', 'last_peered': '2026-02-17T05:15:01.829462+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:01.829462+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:20.893 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:21.110 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:21.110 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1288,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:21.120 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:21.120 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:21.299 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:21.300 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:21.309 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2026-02-17T05:15:20.424993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001695"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084501,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:24.310 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:24.489 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:24.490 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:24.499 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":670,"stamp":"2026-02-17T05:15:24.425454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084501,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:24.500 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:24.674 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:24.675 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:24.683 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":670,"stamp":"2026-02-17T05:15:24.425454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084501,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:24.684 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:24.684 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80133, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:01.829462+0000', 'last_change': '2026-02-17T05:15:01.829462+0000', 'last_active': '2026-02-17T05:15:01.829462+0000', 'last_peered': '2026-02-17T05:15:01.829462+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:01.829462+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:24.684 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:24.907 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:24.907 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1291,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:24.916 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:24.916 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:25.091 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:25.092 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:25.101 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":670,"stamp":"2026-02-17T05:15:24.425454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084501,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:28.103 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:28.279 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:28.279 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:28.287 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":671,"stamp":"2026-02-17T05:15:26.425717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001595"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084502,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:28.288 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:28.462 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:28.462 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:28.470 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":672,"stamp":"2026-02-17T05:15:28.425986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001535"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084502,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:28.471 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:28.471 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80133, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:01.829462+0000', 'last_change': '2026-02-17T05:15:01.829462+0000', 'last_active': '2026-02-17T05:15:01.829462+0000', 'last_peered': '2026-02-17T05:15:01.829462+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:01.829462+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:28.471 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:28.695 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:28.695 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1295,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:28.704 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:28.705 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:28.878 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:28.878 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:28.886 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":672,"stamp":"2026-02-17T05:15:28.425986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001535"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084502,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:31.887 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:32.065 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:32.066 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:32.075 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":673,"stamp":"2026-02-17T05:15:30.426208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001481"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084503,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:32.076 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:32.249 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:32.250 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:32.258 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":673,"stamp":"2026-02-17T05:15:30.426208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001481"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084503,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:32.259 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:32.259 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80133, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:01.829462+0000', 'last_change': '2026-02-17T05:15:01.829462+0000', 'last_active': '2026-02-17T05:15:01.829462+0000', 'last_peered': '2026-02-17T05:15:01.829462+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:01.829462+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:32.260 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:32.479 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:32.479 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1299,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:32.489 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:32.489 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:32.662 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:32.663 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:32.671 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2026-02-17T05:15:32.426449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001455"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80133,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:01.829462+0000","last_change":"2026-02-17T05:15:01.829462+0000","last_active":"2026-02-17T05:15:01.829462+0000","last_peered":"2026-02-17T05:15:01.829462+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:01.829462+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084503,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:35.673 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:35.851 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:35.852 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:35.860 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":675,"stamp":"2026-02-17T05:15:34.426688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001460"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084503,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:35.861 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:36.038 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:36.038 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:36.046 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":675,"stamp":"2026-02-17T05:15:34.426688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001460"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084503,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:36.047 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:36.047 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80137, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:31.829910+0000', 'last_change': '2026-02-17T05:15:31.829910+0000', 'last_active': '2026-02-17T05:15:31.829910+0000', 'last_peered': '2026-02-17T05:15:31.829910+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:31.829910+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:36.047 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:36.271 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:36.271 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1303,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:36.281 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:36.281 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:36.551 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:15:36.550+0000 7f2a44880640 0 --2- 10.20.193.100:0/3837674004 >> [v2:10.20.193.100:3301/0,v1:10.20.193.100:6790/0] conn(0x7f2a4007c1c0 0x7f2a40123750 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=6 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:15:36.656 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:36.657 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:36.665 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":676,"stamp":"2026-02-17T05:15:36.426966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001512"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084504,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:39.667 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:39.840 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:39.841 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:39.848 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2026-02-17T05:15:38.427203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001485"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084504,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:39.850 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:40.027 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:40.028 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:40.036 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2026-02-17T05:15:38.427203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001485"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084504,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:40.037 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:40.037 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80137, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:31.829910+0000', 'last_change': '2026-02-17T05:15:31.829910+0000', 'last_active': '2026-02-17T05:15:31.829910+0000', 'last_peered': '2026-02-17T05:15:31.829910+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:31.829910+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:40.037 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:40.253 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:40.253 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1307,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:40.261 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:40.262 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:40.438 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:40.439 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:40.447 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":678,"stamp":"2026-02-17T05:15:40.427402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001415"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084505,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:43.448 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:43.623 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:43.624 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:43.632 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":679,"stamp":"2026-02-17T05:15:42.427640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001432"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084505,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:43.633 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:43.810 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:43.810 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:43.819 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":679,"stamp":"2026-02-17T05:15:42.427640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001432"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084505,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:43.820 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:43.820 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80137, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:31.829910+0000', 'last_change': '2026-02-17T05:15:31.829910+0000', 'last_active': '2026-02-17T05:15:31.829910+0000', 'last_peered': '2026-02-17T05:15:31.829910+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:31.829910+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:43.820 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:44.036 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:44.036 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1310,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:44.045 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:44.045 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:44.220 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:44.221 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:44.229 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":679,"stamp":"2026-02-17T05:15:42.427640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001432"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084505,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:47.232 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:47.411 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:47.411 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:47.420 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2026-02-17T05:15:46.428112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001424"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084506,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:47.421 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:47.596 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:47.596 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:47.606 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2026-02-17T05:15:46.428112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001424"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084506,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:47.607 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:47.607 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80137, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:31.829910+0000', 'last_change': '2026-02-17T05:15:31.829910+0000', 'last_active': '2026-02-17T05:15:31.829910+0000', 'last_peered': '2026-02-17T05:15:31.829910+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:31.829910+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:47.607 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:47.825 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:47.825 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1314,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":17,"modified":"2026-02-17T05:11:28.394058+0000","services":{}},"progress_events":{}} 2026-02-17T05:15:47.835 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:47.835 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:48.012 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:48.013 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:48.021 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2026-02-17T05:15:46.428112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001424"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084506,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:51.023 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:51.199 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:51.200 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:51.208 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2026-02-17T05:15:50.428700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001497"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:51.209 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:51.386 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:51.387 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:51.395 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2026-02-17T05:15:50.428700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001497"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:51.397 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:51.397 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80137, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:31.829910+0000', 'last_change': '2026-02-17T05:15:31.829910+0000', 'last_active': '2026-02-17T05:15:31.829910+0000', 'last_peered': '2026-02-17T05:15:31.829910+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:31.829910+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:51.398 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:51.617 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:51.618 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1318,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-17T05:15:48.428444+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:15:51.627 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:51.627 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:51.802 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:51.802 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:51.810 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2026-02-17T05:15:50.428700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001497"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:54.812 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:54.987 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:54.987 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:54.996 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":685,"stamp":"2026-02-17T05:15:54.429212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001571"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:54.996 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:55.167 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:55.168 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:55.176 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":685,"stamp":"2026-02-17T05:15:54.429212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001571"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:55.177 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:55.177 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80137, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:31.829910+0000', 'last_change': '2026-02-17T05:15:31.829910+0000', 'last_active': '2026-02-17T05:15:31.829910+0000', 'last_peered': '2026-02-17T05:15:31.829910+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:31.829910+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:55.177 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:55.393 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:55.393 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1322,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-17T05:15:48.428444+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:15:55.402 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:55.403 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:55.577 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:55.578 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:55.587 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":685,"stamp":"2026-02-17T05:15:54.429212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001571"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:58.589 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:58.763 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:58.763 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:58.772 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":687,"stamp":"2026-02-17T05:15:58.429722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001609"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:58.773 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:58.950 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:58.951 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:58.960 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":687,"stamp":"2026-02-17T05:15:58.429722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001609"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:15:58.961 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:15:58.962 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80137, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:31.829910+0000', 'last_change': '2026-02-17T05:15:31.829910+0000', 'last_active': '2026-02-17T05:15:31.829910+0000', 'last_peered': '2026-02-17T05:15:31.829910+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:31.829910+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:15:58.962 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:15:59.182 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:59.182 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1326,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-17T05:15:48.428444+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:15:59.192 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:15:59.192 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:15:59.369 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:15:59.369 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:15:59.378 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":687,"stamp":"2026-02-17T05:15:58.429722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001609"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:11:58.508977+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:11:58.508977+0000","last_peered":"2026-02-17T05:11:58.508977+0000","last_clean":"2026-02-17T05:11:58.508977+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:11:58.508977+0000","last_undegraded":"2026-02-17T05:11:58.508977+0000","last_fullsized":"2026-02-17T05:11:58.508977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:02.380 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:02.557 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:02.558 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:02.567 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":689,"stamp":"2026-02-17T05:16:02.430316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084509,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:02.568 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:02.835 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:16:02.833+0000 7f24a2ffd640 0 --2- 10.20.193.100:0/2628862510 >> [v2:10.20.193.100:3302/0,v1:10.20.193.100:6791/0] conn(0x7f24a4155e90 0x7f24a4176270 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:16:02.946 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:02.947 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:02.957 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":689,"stamp":"2026-02-17T05:16:02.430316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084509,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:02.958 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:02.958 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80137, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:15:31.829910+0000', 'last_change': '2026-02-17T05:15:31.829910+0000', 'last_active': '2026-02-17T05:15:31.829910+0000', 'last_peered': '2026-02-17T05:15:31.829910+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:15:31.829910+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:02.958 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:03.177 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:03.177 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1330,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-17T05:15:48.428444+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:03.186 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:03.187 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:03.360 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:03.361 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:03.369 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":689,"stamp":"2026-02-17T05:16:02.430316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80137,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:15:31.829910+0000","last_change":"2026-02-17T05:15:31.829910+0000","last_active":"2026-02-17T05:15:31.829910+0000","last_peered":"2026-02-17T05:15:31.829910+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:15:31.829910+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084509,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:06.371 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:06.550 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:06.550 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:06.559 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":691,"stamp":"2026-02-17T05:16:06.430915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001703"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80141,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830336+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830336+0000","last_peered":"2026-02-17T05:16:01.830336+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830336+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084510,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:06.560 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:06.735 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:06.735 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:06.743 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":691,"stamp":"2026-02-17T05:16:06.430915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001703"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80141,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830336+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830336+0000","last_peered":"2026-02-17T05:16:01.830336+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830336+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084510,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:06.744 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:06.745 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80141, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:01.830336+0000', 'last_change': '2026-02-17T05:16:01.830336+0000', 'last_active': '2026-02-17T05:16:01.830336+0000', 'last_peered': '2026-02-17T05:16:01.830336+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:01.830336+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:06.745 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:06.964 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:06.964 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1333,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-17T05:15:48.428444+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:06.974 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:06.974 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:07.147 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:07.147 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:07.155 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":691,"stamp":"2026-02-17T05:16:06.430915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001703"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80141,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830336+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830336+0000","last_peered":"2026-02-17T05:16:01.830336+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830336+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084510,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:10.157 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:10.333 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:10.333 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:10.341 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":692,"stamp":"2026-02-17T05:16:08.431162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001673"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80141,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830336+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830336+0000","last_peered":"2026-02-17T05:16:01.830336+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830336+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084510,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:10.343 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:10.518 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:10.518 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:10.527 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":693,"stamp":"2026-02-17T05:16:10.431496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001774"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80141,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830336+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830336+0000","last_peered":"2026-02-17T05:16:01.830336+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830336+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084511,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:10.528 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:10.528 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80141, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:01.830336+0000', 'last_change': '2026-02-17T05:16:01.830336+0000', 'last_active': '2026-02-17T05:16:01.830336+0000', 'last_peered': '2026-02-17T05:16:01.830336+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:01.830336+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:10.529 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:10.753 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:10.754 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1337,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-17T05:15:48.428444+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:10.763 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:10.763 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:10.938 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:10.938 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:10.946 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":693,"stamp":"2026-02-17T05:16:10.431496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001774"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80141,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830336+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830336+0000","last_peered":"2026-02-17T05:16:01.830336+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830336+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084511,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:13.948 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:14.126 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:14.126 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:14.135 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2026-02-17T05:16:12.431747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001751"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80141,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830336+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830336+0000","last_peered":"2026-02-17T05:16:01.830336+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830336+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084511,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:14.136 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:14.314 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:14.314 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:14.323 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2026-02-17T05:16:12.431747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001751"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80141,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830336+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830336+0000","last_peered":"2026-02-17T05:16:01.830336+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830336+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084511,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:14.324 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:14.324 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80141, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:01.830336+0000', 'last_change': '2026-02-17T05:16:01.830336+0000', 'last_active': '2026-02-17T05:16:01.830336+0000', 'last_peered': '2026-02-17T05:16:01.830336+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:01.830336+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:14.324 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:14.539 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:14.539 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1341,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-17T05:15:48.428444+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:14.549 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:14.549 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:14.724 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:14.724 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:14.733 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":695,"stamp":"2026-02-17T05:16:14.432025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084511,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:17.735 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:17.912 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:17.912 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:17.921 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2026-02-17T05:16:16.432294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001727"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084512,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:17.923 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:18.103 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:18.103 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:18.111 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2026-02-17T05:16:16.432294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001727"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084512,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:18.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:18.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80142, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:01.830361+0000', 'last_change': '2026-02-17T05:16:01.830336+0000', 'last_active': '2026-02-17T05:16:01.830361+0000', 'last_peered': '2026-02-17T05:16:01.830361+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:01.830361+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:18.113 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:18.331 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:18.331 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1345,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":19,"modified":"2026-02-17T05:15:48.428444+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:18.340 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:18.340 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:18.517 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:18.518 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:18.527 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":697,"stamp":"2026-02-17T05:16:18.432589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001674"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084512,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:21.528 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:21.702 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:21.703 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:21.712 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2026-02-17T05:16:20.432823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001661"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084513,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:21.713 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:21.888 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:21.888 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:21.896 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2026-02-17T05:16:20.432823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001661"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084513,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:21.897 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:21.897 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80142, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:01.830361+0000', 'last_change': '2026-02-17T05:16:01.830336+0000', 'last_active': '2026-02-17T05:16:01.830361+0000', 'last_peered': '2026-02-17T05:16:01.830361+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:01.830361+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:21.898 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:22.116 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:22.116 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1349,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":20,"modified":"2026-02-17T05:16:18.432617+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:22.126 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:22.126 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:22.303 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:22.303 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:22.311 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2026-02-17T05:16:20.432823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001661"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084513,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:25.313 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:25.491 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:25.491 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:25.500 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":700,"stamp":"2026-02-17T05:16:24.433313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084513,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:25.501 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:25.679 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:25.680 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:25.689 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":700,"stamp":"2026-02-17T05:16:24.433313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084513,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:25.690 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:25.690 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80142, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:01.830361+0000', 'last_change': '2026-02-17T05:16:01.830336+0000', 'last_active': '2026-02-17T05:16:01.830361+0000', 'last_peered': '2026-02-17T05:16:01.830361+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:01.830361+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:25.690 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:25.908 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:25.908 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1352,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-17T05:16:22.433103+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:25.917 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:25.917 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:26.090 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:26.090 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:26.099 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":700,"stamp":"2026-02-17T05:16:24.433313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084513,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:29.101 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:29.277 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:29.278 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:29.286 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":702,"stamp":"2026-02-17T05:16:28.433853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001559"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084514,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:29.287 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:29.461 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:29.462 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:29.470 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":702,"stamp":"2026-02-17T05:16:28.433853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001559"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084514,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:29.471 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:29.472 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80142, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:01.830361+0000', 'last_change': '2026-02-17T05:16:01.830336+0000', 'last_active': '2026-02-17T05:16:01.830361+0000', 'last_peered': '2026-02-17T05:16:01.830361+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:01.830361+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:29.472 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:29.688 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:29.688 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1356,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-17T05:16:22.433103+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:29.698 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:29.698 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:29.874 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:29.875 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:29.883 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":702,"stamp":"2026-02-17T05:16:28.433853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001559"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084514,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:32.884 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:33.059 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:33.059 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:33.068 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":704,"stamp":"2026-02-17T05:16:32.434390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001567"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:33.069 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:33.242 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:33.243 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:33.250 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":704,"stamp":"2026-02-17T05:16:32.434390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001567"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:33.251 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:33.251 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80142, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:01.830361+0000', 'last_change': '2026-02-17T05:16:01.830336+0000', 'last_active': '2026-02-17T05:16:01.830361+0000', 'last_peered': '2026-02-17T05:16:01.830361+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:01.830361+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:33.251 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:33.467 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:33.467 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1360,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-17T05:16:22.433103+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:33.476 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:33.476 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:33.652 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:33.652 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:33.660 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":704,"stamp":"2026-02-17T05:16:32.434390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001567"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80142,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:01.830361+0000","last_change":"2026-02-17T05:16:01.830336+0000","last_active":"2026-02-17T05:16:01.830361+0000","last_peered":"2026-02-17T05:16:01.830361+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:01.830361+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:36.662 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:36.839 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:36.840 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:36.848 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":706,"stamp":"2026-02-17T05:16:36.434942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001629"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:36.850 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:37.024 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:37.024 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:37.032 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":706,"stamp":"2026-02-17T05:16:36.434942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001629"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:37.033 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:37.033 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80146, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:31.830904+0000', 'last_change': '2026-02-17T05:16:31.830904+0000', 'last_active': '2026-02-17T05:16:31.830904+0000', 'last_peered': '2026-02-17T05:16:31.830904+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:31.830904+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:37.034 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:37.251 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:37.251 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1364,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":21,"modified":"2026-02-17T05:16:22.433103+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:37.260 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:37.260 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:37.435 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:37.436 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:37.445 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":706,"stamp":"2026-02-17T05:16:36.434942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001629"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:40.447 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:40.623 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:40.623 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:40.632 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":708,"stamp":"2026-02-17T05:16:40.435545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001691"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:40.633 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:40.811 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:40.811 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:40.819 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":708,"stamp":"2026-02-17T05:16:40.435545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001691"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:40.820 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:40.820 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80146, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:31.830904+0000', 'last_change': '2026-02-17T05:16:31.830904+0000', 'last_active': '2026-02-17T05:16:31.830904+0000', 'last_peered': '2026-02-17T05:16:31.830904+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:31.830904+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:40.821 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:41.038 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:41.038 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1367,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-17T05:16:38.435261+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:41.047 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:41.047 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:41.224 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:41.224 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:41.232 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":708,"stamp":"2026-02-17T05:16:40.435545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001691"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:44.234 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:44.412 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:44.412 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:44.421 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2026-02-17T05:16:42.435816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001725"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:44.422 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:44.595 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:44.596 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:44.604 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":710,"stamp":"2026-02-17T05:16:44.436105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001714"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:44.605 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:44.605 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80146, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:31.830904+0000', 'last_change': '2026-02-17T05:16:31.830904+0000', 'last_active': '2026-02-17T05:16:31.830904+0000', 'last_peered': '2026-02-17T05:16:31.830904+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:31.830904+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:44.606 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:44.822 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:44.823 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1371,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-17T05:16:38.435261+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:44.832 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:44.832 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:45.008 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:45.009 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:45.017 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":710,"stamp":"2026-02-17T05:16:44.436105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001714"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:48.019 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:48.199 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:48.200 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:48.208 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":711,"stamp":"2026-02-17T05:16:46.436362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001729"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:48.209 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:48.383 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:48.384 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:48.393 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":711,"stamp":"2026-02-17T05:16:46.436362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001729"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:48.394 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:48.394 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80146, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:31.830904+0000', 'last_change': '2026-02-17T05:16:31.830904+0000', 'last_active': '2026-02-17T05:16:31.830904+0000', 'last_peered': '2026-02-17T05:16:31.830904+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:31.830904+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:48.395 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:48.664 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:16:48.662+0000 7f6d07fff640 0 --2- 10.20.193.100:0/1690670421 >> v2:10.20.193.100:3300/0 conn(0x7f6d10158380 0x7f6d10178760 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=5616781237479245076 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:16:48.816 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:48.816 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1375,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-17T05:16:38.435261+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:48.827 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:48.827 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:49.002 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:49.003 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:49.011 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":712,"stamp":"2026-02-17T05:16:48.436646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001703"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:52.013 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:52.192 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:52.192 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:52.201 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":713,"stamp":"2026-02-17T05:16:50.436860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001646"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:52.202 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:52.472 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:52.473 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:52.481 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":714,"stamp":"2026-02-17T05:16:52.437173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:52.482 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:52.482 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80146, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:31.830904+0000', 'last_change': '2026-02-17T05:16:31.830904+0000', 'last_active': '2026-02-17T05:16:31.830904+0000', 'last_peered': '2026-02-17T05:16:31.830904+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:31.830904+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:52.483 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:52.700 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:52.701 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1379,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-17T05:16:38.435261+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:52.709 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:52.710 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:52.884 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:52.885 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:52.893 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":714,"stamp":"2026-02-17T05:16:52.437173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:55.895 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:56.071 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:56.072 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:56.080 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2026-02-17T05:16:54.437428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001611"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:56.081 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:56.261 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:56.261 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:56.270 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2026-02-17T05:16:54.437428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001611"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:56.272 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:16:56.272 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80146, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:31.830904+0000', 'last_change': '2026-02-17T05:16:31.830904+0000', 'last_active': '2026-02-17T05:16:31.830904+0000', 'last_peered': '2026-02-17T05:16:31.830904+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:31.830904+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:16:56.272 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:16:56.493 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:56.493 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1383,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-17T05:16:38.435261+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:16:56.503 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:16:56.503 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:56.678 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:56.678 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:56.687 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":716,"stamp":"2026-02-17T05:16:56.437722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084520,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:59.689 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:16:59.961 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:16:59.962 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:16:59.971 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2026-02-17T05:16:58.437942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001580"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084520,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:16:59.971 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:00.150 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:00.150 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:00.158 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2026-02-17T05:16:58.437942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001580"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084520,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:00.160 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:00.160 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80146, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:31.830904+0000', 'last_change': '2026-02-17T05:16:31.830904+0000', 'last_active': '2026-02-17T05:16:31.830904+0000', 'last_peered': '2026-02-17T05:16:31.830904+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:31.830904+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:00.160 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:00.377 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:00.377 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1387,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-17T05:16:38.435261+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:17:00.386 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:00.386 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:00.561 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:00.562 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:00.572 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":718,"stamp":"2026-02-17T05:17:00.438185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001539"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:03.574 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:03.751 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:03.751 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:03.760 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":719,"stamp":"2026-02-17T05:17:02.438493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:03.761 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:03.937 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:03.938 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:03.946 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":719,"stamp":"2026-02-17T05:17:02.438493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:03.947 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:03.947 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80146, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:16:31.830904+0000', 'last_change': '2026-02-17T05:16:31.830904+0000', 'last_active': '2026-02-17T05:16:31.830904+0000', 'last_peered': '2026-02-17T05:16:31.830904+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:16:31.830904+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:03.948 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:04.168 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:04.168 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1391,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-17T05:16:38.435261+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:17:04.176 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:04.177 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:04.352 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:04.353 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:04.360 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":719,"stamp":"2026-02-17T05:17:02.438493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001632"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80146,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:16:31.830904+0000","last_change":"2026-02-17T05:16:31.830904+0000","last_active":"2026-02-17T05:16:31.830904+0000","last_peered":"2026-02-17T05:16:31.830904+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:16:31.830904+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:07.362 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:07.537 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:07.538 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:07.546 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":721,"stamp":"2026-02-17T05:17:06.439039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001610"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084522,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:07.547 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:07.721 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:07.721 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:07.729 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":721,"stamp":"2026-02-17T05:17:06.439039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001610"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084522,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:07.730 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:07.731 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80150, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:01.831457+0000', 'last_change': '2026-02-17T05:17:01.831457+0000', 'last_active': '2026-02-17T05:17:01.831457+0000', 'last_peered': '2026-02-17T05:17:01.831457+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:01.831457+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:07.731 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:07.949 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:07.950 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1394,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-17T05:16:38.435261+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:17:07.961 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:07.961 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:08.140 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:08.140 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:08.149 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":721,"stamp":"2026-02-17T05:17:06.439039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001610"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084522,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:11.150 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:11.328 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:11.328 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:11.336 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":723,"stamp":"2026-02-17T05:17:10.439521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001578"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:11.337 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:11.512 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:11.513 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:11.522 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":723,"stamp":"2026-02-17T05:17:10.439521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001578"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:11.523 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:11.523 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80150, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:01.831457+0000', 'last_change': '2026-02-17T05:17:01.831457+0000', 'last_active': '2026-02-17T05:17:01.831457+0000', 'last_peered': '2026-02-17T05:17:01.831457+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:01.831457+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:11.524 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:11.789 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:17:11.788+0000 7fe72b7fe640 0 --2- 10.20.193.100:0/2813267089 >> v2:10.20.193.100:3300/0 conn(0x7fe72c133e70 0x7fe72c134240 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:17:11.943 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:11.944 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1398,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":22,"modified":"2026-02-17T05:16:38.435261+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-17T05:17:11.953 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:11.953 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:12.222 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:17:12.222+0000 7fa079e2c640 0 --2- 10.20.193.100:0/1178014242 >> [v2:10.20.193.100:3302/0,v1:10.20.193.100:6791/0] conn(0x7fa07407c1c0 0x7fa074126c50 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=1 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:17:12.329 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:12.329 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:12.338 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":723,"stamp":"2026-02-17T05:17:10.439521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001578"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:15.340 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:15.517 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:15.517 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:15.525 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":725,"stamp":"2026-02-17T05:17:14.440078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001584"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:15.525 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:15.700 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:15.700 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:15.710 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":725,"stamp":"2026-02-17T05:17:14.440078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001584"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:15.711 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:15.711 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80150, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:01.831457+0000', 'last_change': '2026-02-17T05:17:01.831457+0000', 'last_active': '2026-02-17T05:17:01.831457+0000', 'last_peered': '2026-02-17T05:17:01.831457+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:01.831457+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:15.712 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:15.929 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:15.930 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1402,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:15.939 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:15.939 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:16.120 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:16.121 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:16.130 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":725,"stamp":"2026-02-17T05:17:14.440078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001584"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:19.131 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:19.309 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:19.310 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:19.318 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":727,"stamp":"2026-02-17T05:17:18.440748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:19.319 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:19.497 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:19.497 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:19.506 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":727,"stamp":"2026-02-17T05:17:18.440748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:19.507 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:19.507 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80150, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:01.831457+0000', 'last_change': '2026-02-17T05:17:01.831457+0000', 'last_active': '2026-02-17T05:17:01.831457+0000', 'last_peered': '2026-02-17T05:17:01.831457+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:01.831457+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:19.507 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:19.735 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:19.736 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1406,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:19.745 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:19.745 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:19.925 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:19.926 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:19.934 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":727,"stamp":"2026-02-17T05:17:18.440748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:22.935 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:23.111 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:23.111 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:23.120 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2026-02-17T05:17:22.441297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001776"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:23.121 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:23.296 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:23.297 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:23.305 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2026-02-17T05:17:22.441297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001776"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:23.306 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:23.306 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80150, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:01.831457+0000', 'last_change': '2026-02-17T05:17:01.831457+0000', 'last_active': '2026-02-17T05:17:01.831457+0000', 'last_peered': '2026-02-17T05:17:01.831457+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:01.831457+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:23.306 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:23.527 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:23.528 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1410,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:23.538 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:23.538 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:23.713 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:23.713 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:23.721 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2026-02-17T05:17:22.441297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001776"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:26.722 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:26.903 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:26.904 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:26.912 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":731,"stamp":"2026-02-17T05:17:26.441735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:26.913 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:27.091 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:27.091 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:27.100 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":731,"stamp":"2026-02-17T05:17:26.441735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:27.101 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:27.101 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80150, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:01.831457+0000', 'last_change': '2026-02-17T05:17:01.831457+0000', 'last_active': '2026-02-17T05:17:01.831457+0000', 'last_peered': '2026-02-17T05:17:01.831457+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:01.831457+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:27.101 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:27.325 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:27.326 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1414,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:27.335 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:27.335 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:27.513 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:27.513 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:27.522 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":731,"stamp":"2026-02-17T05:17:26.441735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:30.524 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:30.703 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:30.704 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:30.713 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":733,"stamp":"2026-02-17T05:17:30.442231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001482"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:30.714 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:30.893 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:30.894 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:30.901 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":733,"stamp":"2026-02-17T05:17:30.442231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001482"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:30.902 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:30.902 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80150, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:01.831457+0000', 'last_change': '2026-02-17T05:17:01.831457+0000', 'last_active': '2026-02-17T05:17:01.831457+0000', 'last_peered': '2026-02-17T05:17:01.831457+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:01.831457+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:30.903 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:31.119 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:31.120 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1418,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:31.130 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:31.130 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:31.311 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:31.311 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:31.320 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":733,"stamp":"2026-02-17T05:17:30.442231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001482"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80150,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:01.831457+0000","last_change":"2026-02-17T05:17:01.831457+0000","last_active":"2026-02-17T05:17:01.831457+0000","last_peered":"2026-02-17T05:17:01.831457+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:01.831457+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:34.323 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:34.501 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:34.502 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:34.510 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":735,"stamp":"2026-02-17T05:17:34.442757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001459"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:34.511 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:34.687 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:34.687 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:34.696 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":735,"stamp":"2026-02-17T05:17:34.442757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001459"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:34.697 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:34.697 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80154, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:31.832033+0000', 'last_change': '2026-02-17T05:17:31.832033+0000', 'last_active': '2026-02-17T05:17:31.832033+0000', 'last_peered': '2026-02-17T05:17:31.832033+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:31.832033+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:34.697 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:34.965 INFO:teuthology.orchestra.run.trial100.stderr:2026-02-17T05:17:34.964+0000 7f041ffff640 0 --2- 10.20.193.100:0/2557127539 >> v2:10.20.193.100:3300/0 conn(0x7f042805c490 0x7f0428137150 unknown :-1 s=AUTH_CONNECTING pgs=0 gs=5 cs=0 l=0 c_cookie=0 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-17T05:17:35.117 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:35.117 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1422,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:35.127 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:35.127 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:35.300 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:35.301 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:35.309 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":735,"stamp":"2026-02-17T05:17:34.442757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001459"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:38.311 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:38.490 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:38.491 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:38.500 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2026-02-17T05:17:38.443317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001581"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084528,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:38.501 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:38.676 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:38.677 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:38.684 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2026-02-17T05:17:38.443317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001581"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084528,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:38.685 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:38.685 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80154, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:31.832033+0000', 'last_change': '2026-02-17T05:17:31.832033+0000', 'last_active': '2026-02-17T05:17:31.832033+0000', 'last_peered': '2026-02-17T05:17:31.832033+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:31.832033+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:38.686 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:38.901 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:38.901 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1425,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:38.911 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:38.911 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:39.087 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:39.087 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:39.096 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2026-02-17T05:17:38.443317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001581"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084528,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:42.098 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:42.273 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:42.274 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:42.282 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":738,"stamp":"2026-02-17T05:17:40.443602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001633"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084529,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:42.283 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:42.457 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:42.457 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:42.466 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":739,"stamp":"2026-02-17T05:17:42.443916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001685"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084529,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:42.467 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:42.467 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80154, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:31.832033+0000', 'last_change': '2026-02-17T05:17:31.832033+0000', 'last_active': '2026-02-17T05:17:31.832033+0000', 'last_peered': '2026-02-17T05:17:31.832033+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:31.832033+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:42.467 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:42.681 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:42.682 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1429,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:42.691 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:42.691 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:42.870 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:42.871 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:42.880 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":739,"stamp":"2026-02-17T05:17:42.443916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001685"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084529,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:45.881 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:46.082 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:46.082 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:46.091 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":740,"stamp":"2026-02-17T05:17:44.444170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001692"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084529,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:46.093 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:46.273 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:46.273 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:46.281 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":740,"stamp":"2026-02-17T05:17:44.444170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001692"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084529,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:46.282 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:46.283 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80154, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:31.832033+0000', 'last_change': '2026-02-17T05:17:31.832033+0000', 'last_active': '2026-02-17T05:17:31.832033+0000', 'last_peered': '2026-02-17T05:17:31.832033+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:31.832033+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:46.283 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:46.512 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:46.512 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1433,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:46.521 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:46.522 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:46.701 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:46.701 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:46.710 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":741,"stamp":"2026-02-17T05:17:46.444497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001740"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:49.712 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:49.888 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:49.888 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:49.896 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":742,"stamp":"2026-02-17T05:17:48.444751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001740"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:49.897 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:50.073 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:50.073 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:50.082 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":742,"stamp":"2026-02-17T05:17:48.444751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001740"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:50.082 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:50.082 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80154, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:31.832033+0000', 'last_change': '2026-02-17T05:17:31.832033+0000', 'last_active': '2026-02-17T05:17:31.832033+0000', 'last_peered': '2026-02-17T05:17:31.832033+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:31.832033+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:50.082 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:50.304 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:50.304 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1437,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:50.313 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:50.314 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:50.488 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:50.488 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:50.496 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":743,"stamp":"2026-02-17T05:17:50.445044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001727"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084531,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:53.498 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:53.679 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:53.680 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:53.688 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2026-02-17T05:17:52.445307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084531,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:53.690 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:53.866 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:53.866 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:53.874 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2026-02-17T05:17:52.445307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084531,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:53.875 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:53.876 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80154, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:31.832033+0000', 'last_change': '2026-02-17T05:17:31.832033+0000', 'last_active': '2026-02-17T05:17:31.832033+0000', 'last_peered': '2026-02-17T05:17:31.832033+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:31.832033+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:53.876 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:54.095 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:54.096 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1441,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:54.106 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:54.107 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:54.284 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:54.284 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:54.292 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2026-02-17T05:17:52.445307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084531,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:57.295 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:57.471 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:57.471 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:57.479 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2026-02-17T05:17:56.445889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001719"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084532,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:57.481 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:57.656 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:57.657 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:57.665 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2026-02-17T05:17:56.445889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001719"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084532,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:17:57.666 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:17:57.666 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80154, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:31.832033+0000', 'last_change': '2026-02-17T05:17:31.832033+0000', 'last_active': '2026-02-17T05:17:31.832033+0000', 'last_peered': '2026-02-17T05:17:31.832033+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:31.832033+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:17:57.667 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:17:57.891 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:57.892 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1444,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:17:57.900 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:17:57.900 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:17:58.077 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:17:58.078 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:17:58.087 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2026-02-17T05:17:56.445889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001719"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:13:59.519892+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:13:59.519892+0000","last_peered":"2026-02-17T05:13:59.519892+0000","last_clean":"2026-02-17T05:13:59.519892+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:13:59.519892+0000","last_undegraded":"2026-02-17T05:13:59.519892+0000","last_fullsized":"2026-02-17T05:13:59.519892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084532,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:01.090 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:01.267 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:01.268 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:01.276 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":748,"stamp":"2026-02-17T05:18:00.446409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001658"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:01.277 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:01.455 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:01.456 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:01.464 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":748,"stamp":"2026-02-17T05:18:00.446409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001658"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:01.465 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:18:01.466 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80154, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:17:31.832033+0000', 'last_change': '2026-02-17T05:17:31.832033+0000', 'last_active': '2026-02-17T05:17:31.832033+0000', 'last_peered': '2026-02-17T05:17:31.832033+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:17:31.832033+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:18:01.466 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:18:01.694 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:01.694 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1448,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:18:01.702 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:18:01.703 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:01.880 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:01.880 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:01.889 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":748,"stamp":"2026-02-17T05:18:00.446409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001658"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80154,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:17:31.832033+0000","last_change":"2026-02-17T05:17:31.832033+0000","last_active":"2026-02-17T05:17:31.832033+0000","last_peered":"2026-02-17T05:17:31.832033+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:17:31.832033+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:04.891 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:05.066 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:05.067 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:05.076 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":750,"stamp":"2026-02-17T05:18:04.446857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80158,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:18:01.832434+0000","last_change":"2026-02-17T05:18:01.832434+0000","last_active":"2026-02-17T05:18:01.832434+0000","last_peered":"2026-02-17T05:18:01.832434+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:18:01.832434+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:05.077 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:05.255 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:05.256 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:05.266 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":750,"stamp":"2026-02-17T05:18:04.446857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80158,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:18:01.832434+0000","last_change":"2026-02-17T05:18:01.832434+0000","last_active":"2026-02-17T05:18:01.832434+0000","last_peered":"2026-02-17T05:18:01.832434+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:18:01.832434+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:05.267 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:18:05.267 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80158, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:18:01.832434+0000', 'last_change': '2026-02-17T05:18:01.832434+0000', 'last_active': '2026-02-17T05:18:01.832434+0000', 'last_peered': '2026-02-17T05:18:01.832434+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:18:01.832434+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:18:05.268 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:18:05.492 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:05.492 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1452,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:18:05.501 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:18:05.501 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:05.677 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:05.678 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:05.686 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":750,"stamp":"2026-02-17T05:18:04.446857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80158,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:18:01.832434+0000","last_change":"2026-02-17T05:18:01.832434+0000","last_active":"2026-02-17T05:18:01.832434+0000","last_peered":"2026-02-17T05:18:01.832434+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:18:01.832434+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:08.687 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:08.872 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:08.872 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:08.880 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":752,"stamp":"2026-02-17T05:18:08.447455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001565"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80158,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:18:01.832434+0000","last_change":"2026-02-17T05:18:01.832434+0000","last_active":"2026-02-17T05:18:01.832434+0000","last_peered":"2026-02-17T05:18:01.832434+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:18:01.832434+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084534,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:08.880 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:09.059 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:09.060 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:09.070 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":752,"stamp":"2026-02-17T05:18:08.447455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001565"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80158,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:18:01.832434+0000","last_change":"2026-02-17T05:18:01.832434+0000","last_active":"2026-02-17T05:18:01.832434+0000","last_peered":"2026-02-17T05:18:01.832434+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:18:01.832434+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084534,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:09.071 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:18:09.071 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80158, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:18:01.832434+0000', 'last_change': '2026-02-17T05:18:01.832434+0000', 'last_active': '2026-02-17T05:18:01.832434+0000', 'last_peered': '2026-02-17T05:18:01.832434+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:18:01.832434+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:18:09.071 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:18:09.291 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:09.291 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1456,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:18:09.300 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:18:09.301 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:09.476 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:09.476 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:09.485 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":752,"stamp":"2026-02-17T05:18:08.447455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001565"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80158,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:18:01.832434+0000","last_change":"2026-02-17T05:18:01.832434+0000","last_active":"2026-02-17T05:18:01.832434+0000","last_peered":"2026-02-17T05:18:01.832434+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:18:01.832434+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084534,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:12.485 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:12.662 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:12.662 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:12.671 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":754,"stamp":"2026-02-17T05:18:12.448032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80158,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:18:01.832434+0000","last_change":"2026-02-17T05:18:01.832434+0000","last_active":"2026-02-17T05:18:01.832434+0000","last_peered":"2026-02-17T05:18:01.832434+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:18:01.832434+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705964,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084535,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:12.672 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:12.844 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:12.844 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:12.853 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":754,"stamp":"2026-02-17T05:18:12.448032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80158,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:18:01.832434+0000","last_change":"2026-02-17T05:18:01.832434+0000","last_active":"2026-02-17T05:18:01.832434+0000","last_peered":"2026-02-17T05:18:01.832434+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:18:01.832434+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705964,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084535,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:12.854 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:18:12.854 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80158, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:18:01.832434+0000', 'last_change': '2026-02-17T05:18:01.832434+0000', 'last_active': '2026-02-17T05:18:01.832434+0000', 'last_peered': '2026-02-17T05:18:01.832434+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:18:01.832434+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:18:12.854 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T05:18:13.074 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:13.074 INFO:teuthology.orchestra.run.trial100.stdout:{"fsid":"ac0ddb8a-2bd5-421d-9307-ad42ed106d65","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"3 backfillfull osd(s)","count":3},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19315/57949 objects degraded (33.331%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1460,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1771304279,"num_in_osds":4,"osd_in_since":1771304034,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":2,"num_pools":2,"num_objects":19317,"data_bytes":81009115672,"bytes_used":145932046336,"bytes_avail":240615010304,"bytes_total":386547056640,"degraded_objects":19315,"degraded_total":57949,"degraded_ratio":0.33331032459576526},"fsmap":{"epoch":1,"btime":"2026-02-17T04:53:53:064030+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":23,"modified":"2026-02-17T05:17:14.440135+0000","services":{}},"progress_events":{}} 2026-02-17T05:18:13.084 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-17T05:18:13.084 INFO:tasks.ceph.ceph_manager.ceph:dumping pgs not clean 2026-02-17T05:18:13.085 DEBUG:teuthology.orchestra.run.trial100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T05:18:13.260 INFO:teuthology.orchestra.run.trial100.stdout: 2026-02-17T05:18:13.260 INFO:teuthology.orchestra.run.trial100.stderr:dumped all 2026-02-17T05:18:13.269 INFO:teuthology.orchestra.run.trial100.stdout:{"pg_ready":true,"pg_map":{"version":754,"stamp":"2026-02-17T05:18:12.448032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81009115672,"num_objects":19317,"num_object_clones":0,"num_object_copies":57949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19317,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":79984,"num_write_kb":203280708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":142511764,"kb_used_data":141196948,"kb_used_omap":59,"kb_used_meta":1314692,"kb_avail":234975596,"statfs":{"total":386547056640,"available":240615010304,"internally_reserved":0,"allocated":144585674752,"data_stored":144584994976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60706,"internal_metadata":1346245342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001622"},"pg_stats":[{"pgid":"2.0","version":"23'79947","reported_seq":80158,"reported_epoch":31,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2026-02-17T05:18:01.832434+0000","last_change":"2026-02-17T05:18:01.832434+0000","last_active":"2026-02-17T05:18:01.832434+0000","last_peered":"2026-02-17T05:18:01.832434+0000","last_clean":"2026-02-17T04:56:20.225664+0000","last_became_active":"2026-02-17T04:58:01.810342+0000","last_became_peered":"2026-02-17T04:58:01.810342+0000","last_unstale":"2026-02-17T05:18:01.832434+0000","last_undegraded":"2026-02-17T04:58:01.453857+0000","last_fullsized":"2026-02-17T04:58:01.451978+0000","mapping_epoch":29,"log_start":"23'79905","ondisk_log_start":"23'79905","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:54:08.275365+0000","last_clean_scrub_stamp":"2026-02-17T04:54:08.275365+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":3000,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":19315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-17T05:15:59.529173+0000","last_change":"2026-02-17T04:58:00.451711+0000","last_active":"2026-02-17T05:15:59.529173+0000","last_peered":"2026-02-17T05:15:59.529173+0000","last_clean":"2026-02-17T05:15:59.529173+0000","last_became_active":"2026-02-17T04:58:00.451597+0000","last_became_peered":"2026-02-17T04:58:00.451597+0000","last_unstale":"2026-02-17T05:15:59.529173+0000","last_undegraded":"2026-02-17T05:15:59.529173+0000","last_fullsized":"2026-02-17T05:15:59.529173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T04:53:59.262250+0000","last_clean_scrub_stamp":"2026-02-17T04:53:59.262250+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T12:53:40.969819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":81008787480,"num_objects":19315,"num_object_clones":0,"num_object_copies":57945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79947,"num_write_kb":203280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144583962624,"data_stored":144583962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705964,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":18,"kb_used_meta":622189,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18549,"internal_metadata":637122443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40177556,"kb_used_data":39555332,"kb_used_omap":16,"kb_used_meta":622191,"kb_avail":54194284,"statfs":{"total":96636764160,"available":55494946816,"internally_reserved":0,"allocated":40504659968,"data_stored":40504491816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":637123743},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084535,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62129112,"kb_used_data":62085704,"kb_used_omap":7,"kb_used_meta":43384,"kb_avail":32242728,"statfs":{"total":96636764160,"available":33016553472,"internally_reserved":0,"allocated":63575760896,"data_stored":63575589160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7659,"internal_metadata":44425749},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":580,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":593920,"data_stored":422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":27573407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63575166976,"data_stored":63575166976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40504397824,"data_stored":40504397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T05:18:13.270 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2026-02-17T05:18:13.270 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "23'79947", 'reported_seq': 80158, 'reported_epoch': 31, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2026-02-17T05:18:01.832434+0000', 'last_change': '2026-02-17T05:18:01.832434+0000', 'last_active': '2026-02-17T05:18:01.832434+0000', 'last_peered': '2026-02-17T05:18:01.832434+0000', 'last_clean': '2026-02-17T04:56:20.225664+0000', 'last_became_active': '2026-02-17T04:58:01.810342+0000', 'last_became_peered': '2026-02-17T04:58:01.810342+0000', 'last_unstale': '2026-02-17T05:18:01.832434+0000', 'last_undegraded': '2026-02-17T04:58:01.453857+0000', 'last_fullsized': '2026-02-17T04:58:01.451978+0000', 'mapping_epoch': 29, 'log_start': "23'79905", 'ondisk_log_start': "23'79905", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'last_clean_scrub_stamp': '2026-02-17T04:54:08.275365+0000', 'objects_scrubbed': 0, 'log_size': 42, 'log_dups_size': 3000, 'ondisk_log_size': 42, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 81008787480, 'num_objects': 19315, 'num_object_clones': 0, 'num_object_copies': 57945, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 19315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19315, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 79947, 'num_write_kb': 203280386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 19315}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T05:18:13.270 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_325850157d309f65985eea9ec9bb9f1d56726fee/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_325850157d309f65985eea9ec9bb9f1d56726fee/qa/tasks/ceph_manager.py", line 2947, in wait_for_clean assert time.time() - start < timeout, \ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: wait_for_clean: failed before timeout expired 2026-02-17T05:18:13.272 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-17T05:18:13.272 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-17T05:18:13.272 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-17T05:18:13.272 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T05:18:13.316 INFO:tasks.ceph.osd.0:Stopped 2026-02-17T05:18:13.316 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-17T05:18:13.316 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T05:18:13.353 INFO:tasks.ceph.osd.1:Stopped 2026-02-17T05:18:13.353 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-17T05:18:13.353 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T05:18:17.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-17T05:18:19.455 INFO:tasks.ceph.osd.2:Stopped 2026-02-17T05:18:19.456 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-17T05:18:19.456 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T05:18:19.510 INFO:tasks.ceph.osd.3:Stopped 2026-02-17T05:18:19.510 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-17T05:18:19.510 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-17T05:18:19.510 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T05:18:19.541 INFO:tasks.ceph.mgr.x:Stopped 2026-02-17T05:18:19.541 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-17T05:18:19.541 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-17T05:18:19.541 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T05:18:19.593 INFO:tasks.ceph.mon.a:Stopped 2026-02-17T05:18:19.593 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2026-02-17T05:18:19.593 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T05:18:19.643 INFO:tasks.ceph.mon.b:Stopped 2026-02-17T05:18:19.644 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2026-02-17T05:18:19.644 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T05:18:19.695 INFO:tasks.ceph.mon.c:Stopped 2026-02-17T05:18:19.695 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-17T05:18:19.696 DEBUG:teuthology.orchestra.run.trial100:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v Error | grep -E -v 'overall HEALTH_' | grep -E -v '\(OBJECT_' | grep -E -v '\(OSDMAP_FLAGS\)' | grep -E -v '\(OSD_' | grep -E -v '\(PG_' | grep -E -v '\(POOL_BACKFILLFULL\)' | grep -E -v '\(POOL_NEARFULL\)' | grep -E -v '\(SLOW_OPS\)' | grep -E -v '\(TOO_FEW_PGS\)' | grep -E -v 'Monitor daemon marked osd\.[[:digit:]]+ down, but it is still running' | grep -E -v 'slow request' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(OSD_SLOW_PING_TIME' | grep -E -v '\(MON_DOWN\)' | grep -E -v '\(MON_NETSPLIT\)' | head -n 1 2026-02-17T05:18:19.762 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial100.front.sepia.ceph.com 2026-02-17T05:18:19.762 DEBUG:teuthology.orchestra.run.trial100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-17T05:18:19.831 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial100.front.sepia.ceph.com 2026-02-17T05:18:19.831 DEBUG:teuthology.orchestra.run.trial100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-17T05:18:19.918 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial100.front.sepia.ceph.com 2026-02-17T05:18:19.918 DEBUG:teuthology.orchestra.run.trial100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-17T05:18:19.985 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial100.front.sepia.ceph.com 2026-02-17T05:18:19.985 DEBUG:teuthology.orchestra.run.trial100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-17T05:18:20.053 INFO:tasks.ceph:Archiving mon data... 2026-02-17T05:18:20.055 DEBUG:teuthology.misc:Transferring archived files from trial100:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_03:53:03-rados-wip-bharath18-testing-2026-02-13-0856-distro-default-trial/50912/data/mon.a.tgz 2026-02-17T05:18:20.055 DEBUG:teuthology.orchestra.run.trial100:> mktemp 2026-02-17T05:18:20.071 INFO:teuthology.orchestra.run.trial100.stdout:/tmp/tmp.LtlkIcpD3U 2026-02-17T05:18:20.072 DEBUG:teuthology.orchestra.run.trial100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.LtlkIcpD3U 2026-02-17T05:18:20.173 DEBUG:teuthology.orchestra.run.trial100:> sudo chmod 0666 /tmp/tmp.LtlkIcpD3U 2026-02-17T05:18:20.254 DEBUG:teuthology.orchestra.remote:trial100:/tmp/tmp.LtlkIcpD3U is 328KB 2026-02-17T05:18:20.315 DEBUG:teuthology.orchestra.run.trial100:> rm -fr /tmp/tmp.LtlkIcpD3U 2026-02-17T05:18:20.331 DEBUG:teuthology.misc:Transferring archived files from trial100:/var/lib/ceph/mon/ceph-b to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_03:53:03-rados-wip-bharath18-testing-2026-02-13-0856-distro-default-trial/50912/data/mon.b.tgz 2026-02-17T05:18:20.331 DEBUG:teuthology.orchestra.run.trial100:> mktemp 2026-02-17T05:18:20.387 INFO:teuthology.orchestra.run.trial100.stdout:/tmp/tmp.qqizdJDcBD 2026-02-17T05:18:20.387 DEBUG:teuthology.orchestra.run.trial100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.qqizdJDcBD 2026-02-17T05:18:20.487 DEBUG:teuthology.orchestra.run.trial100:> sudo chmod 0666 /tmp/tmp.qqizdJDcBD 2026-02-17T05:18:20.568 DEBUG:teuthology.orchestra.remote:trial100:/tmp/tmp.qqizdJDcBD is 331KB 2026-02-17T05:18:20.630 DEBUG:teuthology.orchestra.run.trial100:> rm -fr /tmp/tmp.qqizdJDcBD 2026-02-17T05:18:20.647 DEBUG:teuthology.misc:Transferring archived files from trial100:/var/lib/ceph/mon/ceph-c to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_03:53:03-rados-wip-bharath18-testing-2026-02-13-0856-distro-default-trial/50912/data/mon.c.tgz 2026-02-17T05:18:20.647 DEBUG:teuthology.orchestra.run.trial100:> mktemp 2026-02-17T05:18:20.702 INFO:teuthology.orchestra.run.trial100.stdout:/tmp/tmp.C1180spEMv 2026-02-17T05:18:20.703 DEBUG:teuthology.orchestra.run.trial100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.C1180spEMv 2026-02-17T05:18:20.802 DEBUG:teuthology.orchestra.run.trial100:> sudo chmod 0666 /tmp/tmp.C1180spEMv 2026-02-17T05:18:20.881 DEBUG:teuthology.orchestra.remote:trial100:/tmp/tmp.C1180spEMv is 331KB 2026-02-17T05:18:20.941 DEBUG:teuthology.orchestra.run.trial100:> rm -fr /tmp/tmp.C1180spEMv 2026-02-17T05:18:20.958 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-17T05:18:20.959 DEBUG:teuthology.orchestra.run.trial100:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-17T05:18:21.078 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-17T05:18:21.078 INFO:tasks.ceph:Archiving crash dumps... 2026-02-17T05:18:21.080 DEBUG:teuthology.misc:Transferring archived files from trial100:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_03:53:03-rados-wip-bharath18-testing-2026-02-13-0856-distro-default-trial/50912/remote/trial100/crash 2026-02-17T05:18:21.081 DEBUG:teuthology.orchestra.run.trial100:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-17T05:18:21.143 INFO:tasks.ceph:Compressing logs... 2026-02-17T05:18:21.143 DEBUG:teuthology.orchestra.run.trial100:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81976.log 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92824.log 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97089.log 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr: 32.1% -- replaced with /var/log/ceph/ceph-client.admin.81976.log.gz 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91341.log 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92824.log.gz 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62094.log 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97089.log.gz 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77664.log 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91341.log.gz 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62094.log.gz 2026-02-17T05:18:21.260 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66569.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77664.log.gz 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73940.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98266.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66569.log.gz 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73940.log.gz 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96501.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99981.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98266.log.gz 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96501.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.96501.log.gz -- 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr: /var/log/ceph/ceph-client.admin.67993.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99981.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99981.log.gz 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67993.log.gz 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103117.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103999.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.103117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63334.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103117.log.gz 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75851.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103999.log.gz 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63334.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.53714.log 2026-02-17T05:18:21.261 INFO:teuthology.orchestra.run.trial100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63334.log.gz 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75851.log.gz 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105224.log 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53714.log.gz 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72470.log 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105224.log.gz 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79967.log 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72470.log.gz 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99932.log 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84476.log 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79967.log.gz 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99932.log.gz 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110174.log 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84476.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.84476.log.gz -- 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr: /var/log/ceph/ceph-client.admin.99148.log 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110174.log.gz 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98168.log 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109341.log 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99148.log.gz 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98168.log.gz 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94100.log 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114746.log 2026-02-17T05:18:21.262 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109341.log.gz 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50214.log 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94100.log.gz 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114746.log.gz 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73352.log 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50214.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50214.log.gz 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73352.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.73352.log.gz 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77713.log 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84623.log 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104734.log 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77713.log.gz 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84623.log.gz 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57389.log 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82564.log 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr: 32.9% -- replaced with /var/log/ceph/ceph-client.admin.104734.log.gz 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57389.log.gz 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105322.log 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51377.log 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82564.log.gz 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105322.log.gz 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112686.log 2026-02-17T05:18:21.263 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67111.log 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51377.log.gz 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94884.log 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112686.log.gz 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114207.log 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67111.log.gz 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114599.log 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94884.log.gz 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90557.log 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114207.log.gz 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114599.log.gz 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108704.log 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60771.log 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90557.log.gz 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57438.log 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108704.log.gz 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54400.log 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60771.log.gz 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73205.log 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57438.log.gz 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54400.log.gz 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67895.log 2026-02-17T05:18:21.264 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97334.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73205.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101010.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67895.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65442.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97334.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85603.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101010.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89917.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65442.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93020.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85603.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107625.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89917.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113668.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93020.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73744.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107625.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113668.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62452.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73744.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72372.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54547.log 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62452.log.gz 2026-02-17T05:18:21.265 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72372.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83594.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54547.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84231.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83594.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49407.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65197.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84231.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49407.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59104.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65197.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94443.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91488.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59104.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94443.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80016.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64854.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91488.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80016.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69610.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63922.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64854.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69610.log.gz 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56360.log 2026-02-17T05:18:21.266 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63922.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56360.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66275.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89770.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66275.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78448.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101941.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89770.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78448.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108557.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51528.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101941.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108557.log: 31.7% -- replaced with /var/log/ceph/ceph-client.admin.108557.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62942.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100618.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51528.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62942.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61702.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76782.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100618.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61702.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72176.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84574.log 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76782.log.gz 2026-02-17T05:18:21.267 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72176.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108017.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98462.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84574.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108017.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98021.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67356.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98462.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98021.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85358.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65638.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67356.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85358.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74430.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96648.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65638.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74430.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71878.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93218.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96648.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71878.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61408.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108066.log 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93218.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61408.log.gz 2026-02-17T05:18:21.268 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54253.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103901.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108066.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54253.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66373.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107233.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103901.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54351.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66373.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67552.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107233.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54351.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92922.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99393.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67552.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92922.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87711.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86632.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99393.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87711.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59252.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72911.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86632.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59252.log.gz 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108851.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78497.log 2026-02-17T05:18:21.269 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72911.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108851.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55380.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51457.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78497.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55380.log: 31.9% -- replaced with /var/log/ceph/ceph-client.admin.55380.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84672.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55331.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84672.log: 95.9% -- replaced with /var/log/ceph/ceph-client.admin.51457.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84672.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111546.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103313.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55331.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111546.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108802.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49784.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103313.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108802.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76537.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49784.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54890.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76537.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89329.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54890.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97040.log 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89329.log.gz 2026-02-17T05:18:21.270 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94394.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97040.log.gz 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57095.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94394.log.gz 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61996.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55674.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57095.log.gz 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61996.log.gz 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58859.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55674.log.gz 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67748.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89034.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58859.log.gz 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67748.log.gz 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111644.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89034.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89034.log.gz -5 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.91047.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104146.log 0.0% 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111644.log.gz 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91047.log.gz 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108949.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69659.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104146.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104146.log.gz 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108949.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.108949.log.gz --verbose 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr: -- /var/log/ceph/ceph-client.admin.64217.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69659.log.gz 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111595.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81584.log 2026-02-17T05:18:21.271 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64217.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111595.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48476.log 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69561.log 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81584.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48476.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81094.log 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5/var/log/ceph/ceph-client.admin.69561.log: --verbose -- /var/log/ceph/ceph-client.admin.82123.log 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69561.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79477.log 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81094.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65050.log 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82123.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79477.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78693.log 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107674.log 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65050.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78693.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91292.log 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59546.log 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107674.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91292.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98805.log 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59546.log.gz 2026-02-17T05:18:21.272 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87171.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114844.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98805.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81535.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87171.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114844.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93365.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76488.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81535.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93365.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48427.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105567.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76488.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48427.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.48427.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.103656.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105567.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58908.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103656.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103656.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5/var/log/ceph/ceph-client.admin.58908.log: --verbose -- /var/log/ceph/ceph-client.admin.69365.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58908.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71976.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69365.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112588.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93610.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71976.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112588.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95129.log 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93610.log.gz 2026-02-17T05:18:21.273 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44492.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95129.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78742.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.44492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58369.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44492.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78742.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67944.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58369.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50392.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67944.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49091.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114942.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50392.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81192.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49091.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60673.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114942.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81192.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57683.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79918.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60673.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100422.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57683.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79918.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.79918.log.gz -5 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.78938.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78889.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100422.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78938.log.gz 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81927.log 2026-02-17T05:18:21.274 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78889.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.78889.log.gz -- 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr: /var/log/ceph/ceph-client.admin.93069.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81927.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61261.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74724.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93069.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61261.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70603.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110762.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74724.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70603.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70603.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80065.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110762.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62045.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80065.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78301.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83839.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62045.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78301.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73548.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98707.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83839.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73548.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81437.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58467.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98707.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81437.log.gz 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84917.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107282.log 2026-02-17T05:18:21.275 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58467.log.gz 2026-02-17T05:18:21.276 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84917.log.gz 2026-02-17T05:18:21.276 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100667.log 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107282.log: 32.0% -- replaced with /var/log/ceph/ceph-client.admin.107282.log.gz 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68483.log 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100667.log.gz 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66863.log 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71829.log 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68483.log.gz 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66863.log.gz 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59301.log 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71829.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.71829.log.gz -- 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr: /var/log/ceph/ceph-client.admin.71535.log 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59301.log.gz 2026-02-17T05:18:21.295 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111203.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71535.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49783.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111203.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56654.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49783.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93463.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56654.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111742.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93463.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83103.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111742.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85015.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83103.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84182.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85015.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92726.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84182.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69757.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92726.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50488.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69757.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78546.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50488.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69120.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78546.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66471.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69120.log: 31.7% -- replaced with /var/log/ceph/ceph-client.admin.69120.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110517.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56556.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr: 31.8% -- replaced with /var/log/ceph/ceph-client.admin.66471.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110517.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92677.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55821.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56556.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92677.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113423.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55821.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61163.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113423.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113423.log.gz 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95815.log 2026-02-17T05:18:21.296 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61163.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76831.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95815.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71388.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76831.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87319.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71388.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62844.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87319.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76635.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62844.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70296.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76635.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75067.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70296.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70296.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.37907.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75067.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93855.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph.tmp-client.admin.37907.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.37907.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108508.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93855.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75557.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108508.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87760.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75557.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68973.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87760.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73842.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68973.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68973.log.gz -5 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.112330.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73842.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94933.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112330.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74136.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94933.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75508.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74136.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109733.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75508.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81143.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109733.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106057.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81143.log.gz 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90508.log 2026-02-17T05:18:21.297 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106057.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89378.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90508.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75263.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89378.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88103.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75263.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56213.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88103.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96109.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56213.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110909.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96109.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57144.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110909.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50122.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57144.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53959.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50122.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51230.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53959.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76586.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51230.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78595.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76586.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107037.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78595.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54204.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107037.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68287.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54204.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91439.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68287.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74920.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91439.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74920.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88348.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57340.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88348.log: 31.7% -- replaced with /var/log/ceph/ceph-client.admin.88348.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84770.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77517.log 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57340.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84770.log.gz 2026-02-17T05:18:21.298 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78252.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92187.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77517.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78252.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94737.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57585.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92187.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94737.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83300.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85995.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57585.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83300.log.gzgzip 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68581.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53616.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85995.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68581.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76390.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53616.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108312.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76390.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109929.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108312.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89182.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109929.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78644.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89182.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79085.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78644.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69463.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79085.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63138.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69463.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81878.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63138.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71780.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81878.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114305.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71780.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71290.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114305.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64511.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71290.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110811.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64511.log.gz 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95766.log 2026-02-17T05:18:21.299 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110811.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66079.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95766.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58565.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66079.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66079.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49260.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58565.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75410.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49260.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70799.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99785.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75410.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70799.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105959.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99785.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58026.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105959.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114893.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58026.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50211.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114893.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54106.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50211.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44273.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54106.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72960.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.44273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44273.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95717.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72960.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49979.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95717.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82956.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49979.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104391.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82956.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114158.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104391.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98070.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114158.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57046.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98070.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113325.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97874.log 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57046.log.gz 2026-02-17T05:18:21.300 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53473.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr: 29.6% -- replaced with /var/log/ceph/ceph-client.admin.113325.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97874.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94590.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53473.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77811.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94590.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79624.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77811.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57291.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79624.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86289.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57291.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48525.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86289.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109047.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48525.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113815.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109047.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68336.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113815.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87613.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68336.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112085.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87613.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95962.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112085.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89672.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95962.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72127.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89672.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55576.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72127.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105714.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55576.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48701.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105714.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57536.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48701.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104097.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57536.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102676.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104097.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106351.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102676.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102676.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101157.log 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106351.log.gz 2026-02-17T05:18:21.301 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105420.log 2026-02-17T05:18:21.302 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101157.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101157.log.gz 2026-02-17T05:18:21.302 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96354.log 2026-02-17T05:18:21.302 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105420.log.gz 2026-02-17T05:18:21.302 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84280.log 2026-02-17T05:18:21.302 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96354.log.gz 2026-02-17T05:18:21.302 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56850.log 2026-02-17T05:18:21.302 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84280.log.gz 2026-02-17T05:18:21.302 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55527.log 2026-02-17T05:18:21.302 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56850.log.gz 2026-02-17T05:18:21.302 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54988.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55527.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114403.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54988.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57977.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114403.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101598.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57977.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90214.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101598.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101598.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62599.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90214.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48897.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62599.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57634.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48897.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112281.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57634.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73156.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112281.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87662.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73156.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105861.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87662.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109831.log 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105861.log.gz 2026-02-17T05:18:21.305 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82270.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109831.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62991.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82270.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84035.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62991.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68777.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84035.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88495.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68777.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68777.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97972.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88495.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76292.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97972.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85162.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76292.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101647.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85162.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70051.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101647.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78399.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70051.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93512.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78399.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99099.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92775.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr: 31.7% -- replaced with /var/log/ceph/ceph-client.admin.93512.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99099.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97776.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92775.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100128.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97776.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82760.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100128.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51181.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82760.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68532.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65785.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51181.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68532.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82809.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82074.log 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65785.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82809.log.gz 2026-02-17T05:18:21.306 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107380.log 2026-02-17T05:18:21.307 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82074.log.gz 2026-02-17T05:18:21.307 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56703.log 2026-02-17T05:18:21.307 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107380.log.gz 2026-02-17T05:18:21.307 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82025.log 2026-02-17T05:18:21.307 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56703.log.gz 2026-02-17T05:18:21.307 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84378.log 2026-02-17T05:18:21.307 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82025.log.gz 2026-02-17T05:18:21.307 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93316.log 2026-02-17T05:18:21.307 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84378.log.gz 2026-02-17T05:18:21.307 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70505.log 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93316.log.gz 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96844.log 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70505.log.gz 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113962.log 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96844.log.gz 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77958.log 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113962.log.gz 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103460.log 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77958.log.gz 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56409.log 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104244.log 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103460.log.gz 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56409.log.gz 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98364.log 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104244.log.gz 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113619.log 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98364.log.gz 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50318.log 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113619.log.gz 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108263.log 2026-02-17T05:18:21.308 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50318.log.gz 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101794.log 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108263.log.gz 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113276.log 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101794.log.gz 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64168.log 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113276.log.gz 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78791.log 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64168.log.gz 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53910.log 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78791.log.gz 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83054.log 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53910.log.gz 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48702.log 2026-02-17T05:18:21.309 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83054.log.gz 2026-02-17T05:18:21.310 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91537.log 2026-02-17T05:18:21.310 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48702.log.gz 2026-02-17T05:18:21.310 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114109.log 2026-02-17T05:18:21.310 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91537.log.gz 2026-02-17T05:18:21.310 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113080.log 2026-02-17T05:18:21.310 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114109.log.gz 2026-02-17T05:18:21.310 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105028.log 2026-02-17T05:18:21.310 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113080.log.gz 2026-02-17T05:18:21.310 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88642.log 2026-02-17T05:18:21.311 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105028.log.gz 2026-02-17T05:18:21.311 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64266.log 2026-02-17T05:18:21.311 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88642.log.gz 2026-02-17T05:18:21.311 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104881.log 2026-02-17T05:18:21.311 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64266.log.gz 2026-02-17T05:18:21.311 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98119.log 2026-02-17T05:18:21.311 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104881.log: 30.3% -- replaced with /var/log/ceph/ceph-client.admin.104881.log.gz 2026-02-17T05:18:21.312 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104489.log 2026-02-17T05:18:21.312 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98119.log.gz 2026-02-17T05:18:21.312 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99883.log 2026-02-17T05:18:21.312 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104489.log.gz 2026-02-17T05:18:21.312 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58712.log 2026-02-17T05:18:21.312 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99883.log.gz 2026-02-17T05:18:21.312 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106743.log 2026-02-17T05:18:21.312 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58712.log.gz 2026-02-17T05:18:21.312 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65099.log 2026-02-17T05:18:21.313 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106743.log.gz 2026-02-17T05:18:21.313 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102431.log 2026-02-17T05:18:21.313 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65099.log.gz 2026-02-17T05:18:21.313 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71437.log 2026-02-17T05:18:21.313 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102431.log.gz 2026-02-17T05:18:21.313 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49211.log 2026-02-17T05:18:21.313 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71437.log.gz 2026-02-17T05:18:21.313 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78154.log 2026-02-17T05:18:21.314 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49211.log.gz 2026-02-17T05:18:21.314 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49096.log 2026-02-17T05:18:21.314 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78154.log.gz 2026-02-17T05:18:21.314 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105175.log 2026-02-17T05:18:21.314 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49096.log.gz 2026-02-17T05:18:21.314 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56801.log 2026-02-17T05:18:21.314 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105175.log.gz 2026-02-17T05:18:21.314 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88152.log 2026-02-17T05:18:21.315 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56801.log.gz 2026-02-17T05:18:21.315 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50537.log 2026-02-17T05:18:21.315 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88152.log.gz 2026-02-17T05:18:21.315 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106645.log 2026-02-17T05:18:21.315 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50537.log.gz 2026-02-17T05:18:21.315 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89868.log 2026-02-17T05:18:21.315 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90361.log 2026-02-17T05:18:21.315 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106645.log.gz 2026-02-17T05:18:21.316 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89868.log.gz 2026-02-17T05:18:21.316 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83790.log 2026-02-17T05:18:21.316 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94296.log 2026-02-17T05:18:21.316 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90361.log.gz 2026-02-17T05:18:21.316 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83790.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83790.log.gz 2026-02-17T05:18:21.316 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57879.log 2026-02-17T05:18:21.316 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94296.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94296.log.gz 2026-02-17T05:18:21.316 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65834.log 2026-02-17T05:18:21.317 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57879.log.gz 2026-02-17T05:18:21.317 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54645.log 2026-02-17T05:18:21.317 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96599.log 2026-02-17T05:18:21.317 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65834.log.gz 2026-02-17T05:18:21.317 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54645.log.gz 2026-02-17T05:18:21.317 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68826.log 2026-02-17T05:18:21.317 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72323.log 2026-02-17T05:18:21.318 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96599.log.gz 2026-02-17T05:18:21.318 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68826.log.gz 2026-02-17T05:18:21.318 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101451.log 2026-02-17T05:18:21.318 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72323.log.gz 2026-02-17T05:18:21.318 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67258.log 2026-02-17T05:18:21.318 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101451.log.gz 2026-02-17T05:18:21.318 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69316.log 2026-02-17T05:18:21.318 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67258.log.gz 2026-02-17T05:18:21.318 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83545.log 2026-02-17T05:18:21.319 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69316.log.gz 2026-02-17T05:18:21.319 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63530.log 2026-02-17T05:18:21.319 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83545.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83545.log.gz 2026-02-17T05:18:21.319 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105518.log 2026-02-17T05:18:21.319 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63530.log.gz 2026-02-17T05:18:21.319 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80261.log 2026-02-17T05:18:21.319 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113717.log 2026-02-17T05:18:21.320 INFO:teuthology.orchestra.run.trial100.stderr: 31.4% -- replaced with /var/log/ceph/ceph-client.admin.105518.log.gz 2026-02-17T05:18:21.320 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80261.log.gz 2026-02-17T05:18:21.320 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62795.log 2026-02-17T05:18:21.320 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113717.log.gz 2026-02-17T05:18:21.320 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80310.log 2026-02-17T05:18:21.320 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62795.log.gz 2026-02-17T05:18:21.320 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78007.log 2026-02-17T05:18:21.320 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80310.log.gz 2026-02-17T05:18:21.320 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87024.log 2026-02-17T05:18:21.321 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78007.log.gz 2026-02-17T05:18:21.321 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92481.log 2026-02-17T05:18:21.321 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87024.log.gz 2026-02-17T05:18:21.321 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73107.log 2026-02-17T05:18:21.321 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92481.log.gz 2026-02-17T05:18:21.321 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55723.log 2026-02-17T05:18:21.321 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73107.log.gz 2026-02-17T05:18:21.321 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88250.log 2026-02-17T05:18:21.322 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55723.log.gz 2026-02-17T05:18:21.322 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67699.log 2026-02-17T05:18:21.322 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88250.log.gz 2026-02-17T05:18:21.322 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112882.log 2026-02-17T05:18:21.322 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67699.log.gz 2026-02-17T05:18:21.322 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89525.log 2026-02-17T05:18:21.322 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112882.log.gz 2026-02-17T05:18:21.322 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94345.log 2026-02-17T05:18:21.323 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89525.log.gz 2026-02-17T05:18:21.323 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54449.log 2026-02-17T05:18:21.323 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94345.log.gz 2026-02-17T05:18:21.323 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61359.log 2026-02-17T05:18:21.323 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54449.log.gz 2026-02-17T05:18:21.323 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115187.log 2026-02-17T05:18:21.323 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61359.log.gz 2026-02-17T05:18:21.323 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106694.log 2026-02-17T05:18:21.324 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.115187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115187.log.gz 2026-02-17T05:18:21.324 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71339.log 2026-02-17T05:18:21.324 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106694.log.gz 2026-02-17T05:18:21.324 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97236.log 2026-02-17T05:18:21.324 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71339.log.gz 2026-02-17T05:18:21.324 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59644.log 2026-02-17T05:18:21.324 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97236.log: 32.6% -- replaced with /var/log/ceph/ceph-client.admin.97236.log.gzgzip 2026-02-17T05:18:21.324 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54841.log 2026-02-17T05:18:21.325 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59644.log.gz 2026-02-17T05:18:21.325 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102480.log 2026-02-17T05:18:21.325 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99638.log 2026-02-17T05:18:21.325 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54841.log.gz 2026-02-17T05:18:21.325 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97923.log 2026-02-17T05:18:21.325 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102480.log.gz 2026-02-17T05:18:21.325 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60722.log 2026-02-17T05:18:21.326 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99638.log.gz 2026-02-17T05:18:21.326 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97923.log.gz 2026-02-17T05:18:21.326 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60967.log 2026-02-17T05:18:21.326 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60722.log.gz 2026-02-17T05:18:21.326 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73450.log 2026-02-17T05:18:21.326 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60967.log.gz 2026-02-17T05:18:21.326 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66422.log 2026-02-17T05:18:21.326 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73450.log.gz 2026-02-17T05:18:21.327 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73646.log 2026-02-17T05:18:21.327 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66422.log.gz 2026-02-17T05:18:21.327 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71927.log 2026-02-17T05:18:21.327 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73646.log.gz 2026-02-17T05:18:21.327 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65883.log 2026-02-17T05:18:21.327 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71927.log.gz 2026-02-17T05:18:21.327 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78350.log 2026-02-17T05:18:21.327 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65883.log.gz 2026-02-17T05:18:21.327 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84721.log 2026-02-17T05:18:21.328 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78350.log.gz 2026-02-17T05:18:21.328 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49093.log 2026-02-17T05:18:21.328 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84721.log.gz 2026-02-17T05:18:21.328 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90900.log 2026-02-17T05:18:21.328 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49093.log.gz 2026-02-17T05:18:21.328 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90116.log 2026-02-17T05:18:21.328 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90900.log.gz 2026-02-17T05:18:21.328 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44586.log 2026-02-17T05:18:21.329 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90116.log.gz 2026-02-17T05:18:21.329 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55429.log 2026-02-17T05:18:21.329 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.44586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44586.log.gz 2026-02-17T05:18:21.329 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77762.log 2026-02-17T05:18:21.329 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83349.log 2026-02-17T05:18:21.329 INFO:teuthology.orchestra.run.trial100.stderr: 31.5% -- replaced with /var/log/ceph/ceph-client.admin.55429.log.gz 2026-02-17T05:18:21.329 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77762.log.gz 2026-02-17T05:18:21.329 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105077.log 2026-02-17T05:18:21.329 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83349.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83349.log.gz 2026-02-17T05:18:21.330 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113521.log 2026-02-17T05:18:21.330 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105077.log.gz 2026-02-17T05:18:21.330 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55135.log 2026-02-17T05:18:21.330 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113521.log.gz 2026-02-17T05:18:21.330 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77909.log 2026-02-17T05:18:21.330 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55135.log.gz 2026-02-17T05:18:21.330 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107576.log 2026-02-17T05:18:21.330 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77909.log.gz 2026-02-17T05:18:21.331 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106939.log 2026-02-17T05:18:21.331 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107576.log.gz 2026-02-17T05:18:21.331 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59202.log 2026-02-17T05:18:21.331 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106939.log.gz 2026-02-17T05:18:21.331 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91844.log 2026-02-17T05:18:21.331 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59202.log.gz 2026-02-17T05:18:21.331 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54155.log 2026-02-17T05:18:21.331 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91844.log.gz 2026-02-17T05:18:21.332 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94149.log 2026-02-17T05:18:21.332 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54155.log.gz 2026-02-17T05:18:21.332 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54939.log 2026-02-17T05:18:21.332 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94149.log.gz 2026-02-17T05:18:21.332 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66324.log 2026-02-17T05:18:21.332 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54939.log.gz 2026-02-17T05:18:21.332 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73303.log 2026-02-17T05:18:21.332 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66324.log.gz 2026-02-17T05:18:21.332 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81486.log 2026-02-17T05:18:21.333 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73303.log.gz 2026-02-17T05:18:21.333 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72519.log 2026-02-17T05:18:21.333 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81486.log.gz 2026-02-17T05:18:21.333 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104930.log 2026-02-17T05:18:21.333 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72519.log.gz 2026-02-17T05:18:21.333 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55870.log 2026-02-17T05:18:21.333 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104930.log: 31.7% -- replaced with /var/log/ceph/ceph-client.admin.104930.log.gz 2026-02-17T05:18:21.333 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67650.log 2026-02-17T05:18:21.334 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55870.log.gz 2026-02-17T05:18:21.334 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62550.log 2026-02-17T05:18:21.334 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53474.log 2026-02-17T05:18:21.334 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67650.log.gz 2026-02-17T05:18:21.334 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62550.log.gz 2026-02-17T05:18:21.334 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70897.log 2026-02-17T05:18:21.334 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53474.log.gz 2026-02-17T05:18:21.334 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86926.log 2026-02-17T05:18:21.335 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70897.log.gz 2026-02-17T05:18:21.335 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101843.log 2026-02-17T05:18:21.335 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86926.log.gz 2026-02-17T05:18:21.335 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101990.log 2026-02-17T05:18:21.335 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101843.log.gz 2026-02-17T05:18:21.335 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56605.log 2026-02-17T05:18:21.335 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101990.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101990.log.gz 2026-02-17T05:18:21.335 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93414.log 2026-02-17T05:18:21.335 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56605.log.gz 2026-02-17T05:18:21.336 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107184.log 2026-02-17T05:18:21.336 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93414.log: 31.8% -- replaced with /var/log/ceph/ceph-client.admin.93414.log.gz 2026-02-17T05:18:21.336 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64413.log 2026-02-17T05:18:21.336 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107184.log.gz 2026-02-17T05:18:21.336 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74479.log 2026-02-17T05:18:21.336 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64413.log.gz 2026-02-17T05:18:21.336 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93659.log 2026-02-17T05:18:21.336 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74479.log.gz 2026-02-17T05:18:21.337 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64903.log 2026-02-17T05:18:21.337 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93659.log.gz 2026-02-17T05:18:21.337 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51400.log 2026-02-17T05:18:21.337 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64903.log.gz 2026-02-17T05:18:21.337 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113766.log 2026-02-17T05:18:21.337 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54596.log 2026-02-17T05:18:21.337 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113766.log: 96.8% -- replaced with /var/log/ceph/ceph-client.admin.51400.log.gz 2026-02-17T05:18:21.338 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113766.log.gz 2026-02-17T05:18:21.338 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79379.log 2026-02-17T05:18:21.338 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54596.log.gz 2026-02-17T05:18:21.338 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105910.log 2026-02-17T05:18:21.338 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79379.log.gz 2026-02-17T05:18:21.338 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80751.log 2026-02-17T05:18:21.338 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105910.log.gz 2026-02-17T05:18:21.338 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109243.log 2026-02-17T05:18:21.338 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80751.log.gz 2026-02-17T05:18:21.339 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82221.log 2026-02-17T05:18:21.339 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109243.log.gz 2026-02-17T05:18:21.339 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69855.log 2026-02-17T05:18:21.339 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82221.log.gz 2026-02-17T05:18:21.339 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74038.log 2026-02-17T05:18:21.339 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69855.log.gz 2026-02-17T05:18:21.339 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66226.log 2026-02-17T05:18:21.339 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74038.log.gz 2026-02-17T05:18:21.339 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74675.log 2026-02-17T05:18:21.340 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66226.log.gz 2026-02-17T05:18:21.340 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109390.log 2026-02-17T05:18:21.340 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74675.log: 30.2% -- replaced with /var/log/ceph/ceph-client.admin.74675.log.gzgzip 2026-02-17T05:18:21.340 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.88936.log 2026-02-17T05:18:21.340 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109390.log.gz 2026-02-17T05:18:21.340 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94247.log 2026-02-17T05:18:21.340 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88936.log: 31.2% -- replaced with /var/log/ceph/ceph-client.admin.88936.log.gz 2026-02-17T05:18:21.340 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99687.log 2026-02-17T05:18:21.341 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94247.log.gz 2026-02-17T05:18:21.341 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79526.log 2026-02-17T05:18:21.341 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99687.log.gz 2026-02-17T05:18:21.341 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112784.log 2026-02-17T05:18:21.341 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79526.log.gz 2026-02-17T05:18:21.341 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87809.log 2026-02-17T05:18:21.341 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112784.log.gz 2026-02-17T05:18:21.341 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54008.log 2026-02-17T05:18:21.342 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87809.log.gz 2026-02-17T05:18:21.342 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95570.log 2026-02-17T05:18:21.342 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54008.log.gz 2026-02-17T05:18:21.342 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110076.log 2026-02-17T05:18:21.342 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95570.log.gz 2026-02-17T05:18:21.342 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91746.log 2026-02-17T05:18:21.342 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110076.log.gz 2026-02-17T05:18:21.342 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67454.log 2026-02-17T05:18:21.343 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91746.log.gz 2026-02-17T05:18:21.343 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105616.log 2026-02-17T05:18:21.343 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67454.log: gzip -5 --verbose -- 32.0% /var/log/ceph/ceph-client.admin.71633.log 2026-02-17T05:18:21.343 INFO:teuthology.orchestra.run.trial100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67454.log.gz 2026-02-17T05:18:21.343 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105616.log.gz 2026-02-17T05:18:21.343 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80408.log 2026-02-17T05:18:21.343 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71633.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71633.log.gz -5 2026-02-17T05:18:21.343 INFO:teuthology.orchestra.run.trial100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.61555.log 2026-02-17T05:18:21.344 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80408.log.gz 2026-02-17T05:18:21.344 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48896.log 2026-02-17T05:18:21.344 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61555.log.gz 2026-02-17T05:18:21.344 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83692.log 2026-02-17T05:18:21.344 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48896.log.gz 2026-02-17T05:18:21.344 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62648.log 2026-02-17T05:18:21.344 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83692.log.gz 2026-02-17T05:18:21.344 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103068.log 2026-02-17T05:18:21.345 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62648.log.gz 2026-02-17T05:18:21.345 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57732.log 2026-02-17T05:18:21.345 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103068.log.gz 2026-02-17T05:18:21.345 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89623.log 2026-02-17T05:18:21.345 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57732.log: 32.0% -- replaced with /var/log/ceph/ceph-client.admin.57732.log.gz 2026-02-17T05:18:21.345 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69022.log 2026-02-17T05:18:21.345 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89623.log.gz 2026-02-17T05:18:21.345 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110860.log 2026-02-17T05:18:21.346 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69022.log: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-17T05:18:21.346 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69022.log.gz 2026-02-17T05:18:21.346 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70652.log 2026-02-17T05:18:21.346 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110860.log.gz 2026-02-17T05:18:21.346 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72225.log 2026-02-17T05:18:21.346 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70652.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.70652.log.gz 2026-02-17T05:18:21.346 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.114648.log 2026-02-17T05:18:21.346 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72225.log.gz 2026-02-17T05:18:21.346 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91390.log 2026-02-17T05:18:21.347 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114648.log.gz 2026-02-17T05:18:21.347 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60183.log 2026-02-17T05:18:21.347 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91390.log.gz 2026-02-17T05:18:21.347 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102039.log 2026-02-17T05:18:21.347 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60183.log.gz 2026-02-17T05:18:21.347 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76439.log 2026-02-17T05:18:21.347 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102039.log.gz 2026-02-17T05:18:21.347 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59006.log 2026-02-17T05:18:21.348 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76439.log.gz 2026-02-17T05:18:21.348 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109194.log 2026-02-17T05:18:21.348 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59006.log.gz 2026-02-17T05:18:21.348 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74283.log 2026-02-17T05:18:21.348 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109194.log.gz 2026-02-17T05:18:21.348 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81045.log 2026-02-17T05:18:21.348 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74283.log.gz 2026-02-17T05:18:21.348 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58075.log 2026-02-17T05:18:21.348 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81045.log.gz 2026-02-17T05:18:21.349 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68875.log 2026-02-17T05:18:21.349 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58075.log.gz 2026-02-17T05:18:21.349 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86044.log 2026-02-17T05:18:21.349 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68875.log.gz 2026-02-17T05:18:21.349 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90015.log 2026-02-17T05:18:21.349 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86044.log.gz 2026-02-17T05:18:21.349 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90998.log 2026-02-17T05:18:21.350 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90015.log.gz 2026-02-17T05:18:21.350 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49655.log 2026-02-17T05:18:21.350 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90998.log.gz 2026-02-17T05:18:21.350 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58320.log 2026-02-17T05:18:21.350 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49655.log.gz 2026-02-17T05:18:21.350 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90606.log 2026-02-17T05:18:21.350 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58320.log.gz 2026-02-17T05:18:21.350 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61212.log 2026-02-17T05:18:21.351 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90606.log.gz 2026-02-17T05:18:21.351 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97285.log 2026-02-17T05:18:21.351 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61212.log.gz 2026-02-17T05:18:21.351 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106204.log 2026-02-17T05:18:21.351 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97285.log.gz 2026-02-17T05:18:21.351 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60624.log 2026-02-17T05:18:21.351 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106204.log.gz 2026-02-17T05:18:21.351 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101304.log 2026-02-17T05:18:21.351 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60624.log.gz 2026-02-17T05:18:21.352 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80359.log 2026-02-17T05:18:21.352 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101304.log.gz 2026-02-17T05:18:21.352 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92432.log 2026-02-17T05:18:21.352 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80359.log.gz 2026-02-17T05:18:21.352 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74185.log 2026-02-17T05:18:21.352 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92432.log.gz 2026-02-17T05:18:21.352 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73499.log 2026-02-17T05:18:21.352 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74185.log.gz 2026-02-17T05:18:21.353 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80555.log 2026-02-17T05:18:21.353 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73499.log.gz 2026-02-17T05:18:21.353 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87417.log 2026-02-17T05:18:21.353 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80555.log.gz 2026-02-17T05:18:21.353 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99442.log 2026-02-17T05:18:21.353 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87417.log.gz 2026-02-17T05:18:21.353 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114795.log 2026-02-17T05:18:21.353 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99442.log.gz 2026-02-17T05:18:21.354 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106841.log 2026-02-17T05:18:21.354 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114795.log.gz 2026-02-17T05:18:21.354 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74773.log 2026-02-17T05:18:21.354 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106841.log.gz 2026-02-17T05:18:21.354 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114697.log 2026-02-17T05:18:21.354 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74773.log.gz 2026-02-17T05:18:21.354 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92138.log 2026-02-17T05:18:21.354 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114697.log.gz 2026-02-17T05:18:21.354 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49095.log 2026-02-17T05:18:21.355 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92138.log.gz 2026-02-17T05:18:21.355 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83888.log 2026-02-17T05:18:21.355 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49095.log.gz 2026-02-17T05:18:21.355 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67010.log 2026-02-17T05:18:21.355 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83888.log.gz 2026-02-17T05:18:21.355 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67059.log 2026-02-17T05:18:21.355 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67010.log.gz 2026-02-17T05:18:21.355 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101402.log 2026-02-17T05:18:21.356 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67059.log.gz 2026-02-17T05:18:21.356 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65491.log 2026-02-17T05:18:21.356 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101402.log.gz 2026-02-17T05:18:21.356 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65589.log 2026-02-17T05:18:21.356 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65491.log.gz 2026-02-17T05:18:21.356 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64756.log 2026-02-17T05:18:21.356 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65589.log.gz 2026-02-17T05:18:21.356 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79869.log 2026-02-17T05:18:21.357 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64756.log.gz 2026-02-17T05:18:21.357 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56507.log 2026-02-17T05:18:21.357 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79869.log.gz 2026-02-17T05:18:21.357 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78203.log 2026-02-17T05:18:21.357 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56507.log.gz 2026-02-17T05:18:21.357 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91194.log 2026-02-17T05:18:21.357 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78203.log.gz 2026-02-17T05:18:21.358 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65540.log 2026-02-17T05:18:21.358 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91194.log.gz 2026-02-17T05:18:21.358 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85064.log 2026-02-17T05:18:21.358 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65540.log.gz 2026-02-17T05:18:21.358 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109782.log 2026-02-17T05:18:21.358 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85064.log.gz 2026-02-17T05:18:21.358 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71486.log 2026-02-17T05:18:21.358 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109782.log.gz 2026-02-17T05:18:21.358 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87368.log 2026-02-17T05:18:21.359 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71486.log.gz 2026-02-17T05:18:21.359 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94002.log 2026-02-17T05:18:21.359 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87368.log.gz 2026-02-17T05:18:21.359 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102970.log 2026-02-17T05:18:21.359 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94002.log.gz 2026-02-17T05:18:21.359 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85750.log 2026-02-17T05:18:21.359 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102970.log.gz 2026-02-17T05:18:21.359 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44633.log 2026-02-17T05:18:21.360 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85750.log.gz 2026-02-17T05:18:21.360 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85946.log 2026-02-17T05:18:21.360 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.44633.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.44633.log.gz --verbose -- 2026-02-17T05:18:21.360 INFO:teuthology.orchestra.run.trial100.stderr: /var/log/ceph/ceph-client.admin.79820.log 2026-02-17T05:18:21.360 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85946.log.gz 2026-02-17T05:18:21.360 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64315.log 2026-02-17T05:18:21.360 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79820.log.gz 2026-02-17T05:18:21.360 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86240.log 2026-02-17T05:18:21.361 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64315.log.gz 2026-02-17T05:18:21.361 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76978.log 2026-02-17T05:18:21.361 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86240.log.gz 2026-02-17T05:18:21.361 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99834.log 2026-02-17T05:18:21.361 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76978.log.gz 2026-02-17T05:18:21.361 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110027.log 2026-02-17T05:18:21.361 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99834.log.gz 2026-02-17T05:18:21.361 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51626.log 2026-02-17T05:18:21.361 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110027.log.gz 2026-02-17T05:18:21.362 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100814.log 2026-02-17T05:18:21.362 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51626.log.gz 2026-02-17T05:18:21.362 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59595.log 2026-02-17T05:18:21.362 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100814.log.gz 2026-02-17T05:18:21.362 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103215.log 2026-02-17T05:18:21.362 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59595.log.gz 2026-02-17T05:18:21.362 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109292.log 2026-02-17T05:18:21.362 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103215.log.gz 2026-02-17T05:18:21.363 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65393.log 2026-02-17T05:18:21.363 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109292.log.gz 2026-02-17T05:18:21.363 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114991.log 2026-02-17T05:18:21.363 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65393.log.gz 2026-02-17T05:18:21.363 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101206.log 2026-02-17T05:18:21.363 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114991.log.gz 2026-02-17T05:18:21.363 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71192.log 2026-02-17T05:18:21.363 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101206.log.gz 2026-02-17T05:18:21.363 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105763.log 2026-02-17T05:18:21.364 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71192.log.gz 2026-02-17T05:18:21.364 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107870.log 2026-02-17T05:18:21.364 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105763.log.gz 2026-02-17T05:18:21.364 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67307.log 2026-02-17T05:18:21.364 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107870.log.gz 2026-02-17T05:18:21.364 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61065.log 2026-02-17T05:18:21.364 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67307.log.gz 2026-02-17T05:18:21.364 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111889.log 2026-02-17T05:18:21.365 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61065.log.gz 2026-02-17T05:18:21.365 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107086.log 2026-02-17T05:18:21.365 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111889.log.gz 2026-02-17T05:18:21.365 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97187.log 2026-02-17T05:18:21.365 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107086.log.gz 2026-02-17T05:18:21.365 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69708.log 2026-02-17T05:18:21.365 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97187.log.gz 2026-02-17T05:18:21.366 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92089.log 2026-02-17T05:18:21.366 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69708.log.gz 2026-02-17T05:18:21.366 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86191.log 2026-02-17T05:18:21.366 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92089.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.92530.log 2026-02-17T05:18:21.366 INFO:teuthology.orchestra.run.trial100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92089.log.gz 2026-02-17T05:18:21.366 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86191.log.gz 2026-02-17T05:18:21.366 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98609.log 2026-02-17T05:18:21.366 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92530.log.gz 2026-02-17T05:18:21.367 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104587.log 2026-02-17T05:18:21.367 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98609.log.gz 2026-02-17T05:18:21.367 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111350.log 2026-02-17T05:18:21.367 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104587.log.gz 2026-02-17T05:18:21.367 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92236.log 2026-02-17T05:18:21.367 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111350.log.gz 2026-02-17T05:18:21.367 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104832.log 2026-02-17T05:18:21.367 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92236.log.gz 2026-02-17T05:18:21.368 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76880.log 2026-02-17T05:18:21.368 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104832.log.gz 2026-02-17T05:18:21.368 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80506.log 2026-02-17T05:18:21.368 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76880.log.gz 2026-02-17T05:18:21.368 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74528.log 2026-02-17T05:18:21.368 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80506.log.gz 2026-02-17T05:18:21.368 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95276.log 2026-02-17T05:18:21.368 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74528.log.gz 2026-02-17T05:18:21.368 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68728.log 2026-02-17T05:18:21.369 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95276.log.gz 2026-02-17T05:18:21.369 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99246.log 2026-02-17T05:18:21.369 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68728.log.gz 2026-02-17T05:18:21.369 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113913.log 2026-02-17T05:18:21.369 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99246.log.gz 2026-02-17T05:18:21.369 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91648.log 2026-02-17T05:18:21.369 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113913.log.gz 2026-02-17T05:18:21.369 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104538.log 2026-02-17T05:18:21.370 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91648.log.gz 2026-02-17T05:18:21.370 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99540.log 2026-02-17T05:18:21.370 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104538.log.gz 2026-02-17T05:18:21.370 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86681.log 2026-02-17T05:18:21.370 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99540.log.gz 2026-02-17T05:18:21.370 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96746.log 2026-02-17T05:18:21.370 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86681.log.gz 2026-02-17T05:18:21.371 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53472.log 2026-02-17T05:18:21.371 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96746.log.gz 2026-02-17T05:18:21.371 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92040.log 2026-02-17T05:18:21.371 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53472.log.gz 2026-02-17T05:18:21.371 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62143.log 2026-02-17T05:18:21.371 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92040.log.gz 2026-02-17T05:18:21.371 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80996.log 2026-02-17T05:18:21.371 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62143.log.gz 2026-02-17T05:18:21.372 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96697.log 2026-02-17T05:18:21.372 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80996.log.gz 2026-02-17T05:18:21.372 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97481.log 2026-02-17T05:18:21.372 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96697.log.gz 2026-02-17T05:18:21.372 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91893.log 2026-02-17T05:18:21.372 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97481.log.gz 2026-02-17T05:18:21.372 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72862.log 2026-02-17T05:18:21.372 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91893.log.gz 2026-02-17T05:18:21.372 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80653.log 2026-02-17T05:18:21.373 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72862.log.gz 2026-02-17T05:18:21.373 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57193.log 2026-02-17T05:18:21.373 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80653.log.gz 2026-02-17T05:18:21.373 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55968.log 2026-02-17T05:18:21.373 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57193.log.gz 2026-02-17T05:18:21.373 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95325.log 2026-02-17T05:18:21.373 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55968.log.gz 2026-02-17T05:18:21.373 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64069.log 2026-02-17T05:18:21.374 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95325.log.gz 2026-02-17T05:18:21.374 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53276.log 2026-02-17T05:18:21.374 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64069.log.gz 2026-02-17T05:18:21.374 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88446.log 2026-02-17T05:18:21.374 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53276.log.gz 2026-02-17T05:18:21.374 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103705.log 2026-02-17T05:18:21.374 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88446.log.gz 2026-02-17T05:18:21.374 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102137.log 2026-02-17T05:18:21.375 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103705.log.gz 2026-02-17T05:18:21.375 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52913.log 2026-02-17T05:18:21.375 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102137.log.gz 2026-02-17T05:18:21.375 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94639.log 2026-02-17T05:18:21.375 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.52913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52913.log.gz 2026-02-17T05:18:21.375 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70394.log 2026-02-17T05:18:21.375 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94639.log.gz 2026-02-17T05:18:21.375 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92579.log 2026-02-17T05:18:21.375 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70394.log.gz 2026-02-17T05:18:21.376 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107331.log 2026-02-17T05:18:21.376 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92579.log.gz 2026-02-17T05:18:21.376 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94492.log 2026-02-17T05:18:21.376 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107331.log.gz 2026-02-17T05:18:21.376 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108116.log 2026-02-17T05:18:21.376 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94492.log.gz 2026-02-17T05:18:21.376 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86730.log 2026-02-17T05:18:21.376 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108116.log.gz 2026-02-17T05:18:21.377 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100079.log 2026-02-17T05:18:21.377 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86730.log.gz 2026-02-17T05:18:21.377 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110125.log 2026-02-17T05:18:21.377 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100079.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100079.log.gz 2026-02-17T05:18:21.377 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82368.log 2026-02-17T05:18:21.377 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108753.log 2026-02-17T05:18:21.377 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110125.log.gz 2026-02-17T05:18:21.378 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82368.log.gz 2026-02-17T05:18:21.378 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113864.log 2026-02-17T05:18:21.378 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58957.log 2026-02-17T05:18:21.378 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108753.log.gz 2026-02-17T05:18:21.378 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113864.log.gz 2026-02-17T05:18:21.378 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49358.log 2026-02-17T05:18:21.378 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95423.log 2026-02-17T05:18:21.378 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58957.log.gz 2026-02-17T05:18:21.379 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49358.log.gz 2026-02-17T05:18:21.379 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49782.log 2026-02-17T05:18:21.379 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56115.log 2026-02-17T05:18:21.379 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95423.log.gz 2026-02-17T05:18:21.379 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49782.log.gz 2026-02-17T05:18:21.379 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75655.log 2026-02-17T05:18:21.379 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89133.log 2026-02-17T05:18:21.379 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56115.log.gz 2026-02-17T05:18:21.380 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75655.log.gz 2026-02-17T05:18:21.380 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66814.log 2026-02-17T05:18:21.380 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89133.log.gz 2026-02-17T05:18:21.380 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64560.log 2026-02-17T05:18:21.380 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49505.log 2026-02-17T05:18:21.380 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66814.log.gz 2026-02-17T05:18:21.380 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64560.log.gz 2026-02-17T05:18:21.381 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60379.log 2026-02-17T05:18:21.381 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111252.log 2026-02-17T05:18:21.381 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49505.log.gz 2026-02-17T05:18:21.381 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60379.log.gz 2026-02-17T05:18:21.381 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82613.log 2026-02-17T05:18:21.381 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99197.log 2026-02-17T05:18:21.381 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111252.log.gz 2026-02-17T05:18:21.381 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82613.log.gz 2026-02-17T05:18:21.382 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114452.log 2026-02-17T05:18:21.382 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101353.log 2026-02-17T05:18:21.382 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99197.log.gz 2026-02-17T05:18:21.382 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114452.log.gz 2026-02-17T05:18:21.382 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61849.log 2026-02-17T05:18:21.382 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75802.log 2026-02-17T05:18:21.382 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101353.log.gz 2026-02-17T05:18:21.383 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61849.log.gz 2026-02-17T05:18:21.383 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102088.log 2026-02-17T05:18:21.383 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91096.log 2026-02-17T05:18:21.383 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75802.log.gz 2026-02-17T05:18:21.383 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102088.log.gz 2026-02-17T05:18:21.383 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55233.log 2026-02-17T05:18:21.383 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107527.log 2026-02-17T05:18:21.383 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91096.log.gz 2026-02-17T05:18:21.384 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55233.log: 31.8% -- replaced with /var/log/ceph/ceph-client.admin.55233.log.gz 2026-02-17T05:18:21.384 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44345.log 2026-02-17T05:18:21.384 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89966.log 2026-02-17T05:18:21.384 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107527.log.gz 2026-02-17T05:18:21.384 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.44345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44345.log.gz 2026-02-17T05:18:21.384 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76096.log 2026-02-17T05:18:21.384 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89966.log.gz 2026-02-17T05:18:21.384 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61457.log 2026-02-17T05:18:21.385 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76096.log.gz 2026-02-17T05:18:21.385 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112441.log 2026-02-17T05:18:21.385 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61653.log 2026-02-17T05:18:21.385 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61457.log.gz 2026-02-17T05:18:21.385 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112441.log.gz 2026-02-17T05:18:21.385 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69169.log 2026-02-17T05:18:21.385 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65736.log 2026-02-17T05:18:21.385 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61653.log.gz 2026-02-17T05:18:21.386 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69169.log.gz 2026-02-17T05:18:21.386 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110713.log 2026-02-17T05:18:21.386 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65736.log.gz 2026-02-17T05:18:21.386 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79673.log 2026-02-17T05:18:21.386 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110713.log.gz 2026-02-17T05:18:21.386 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53081.log 2026-02-17T05:18:21.386 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79673.log.gz 2026-02-17T05:18:21.386 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89574.log 2026-02-17T05:18:21.387 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53081.log.gz 2026-02-17T05:18:21.387 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53861.log 2026-02-17T05:18:21.387 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89574.log.gz 2026-02-17T05:18:21.387 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73009.log 2026-02-17T05:18:21.387 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53861.log.gz 2026-02-17T05:18:21.387 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76684.log 2026-02-17T05:18:21.388 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86779.log 2026-02-17T05:18:21.388 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73009.log.gz 2026-02-17T05:18:21.388 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76684.log.gz 2026-02-17T05:18:21.388 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81780.log 2026-02-17T05:18:21.388 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52912.log 2026-02-17T05:18:21.388 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86779.log.gz 2026-02-17T05:18:21.388 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81780.log.gz 2026-02-17T05:18:21.388 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113227.log 2026-02-17T05:18:21.389 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.52912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106988.log 2026-02-17T05:18:21.389 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52912.log.gz 2026-02-17T05:18:21.389 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113227.log.gz 2026-02-17T05:18:21.389 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56997.log 2026-02-17T05:18:21.389 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108459.log 2026-02-17T05:18:21.389 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106988.log.gz 2026-02-17T05:18:21.389 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56997.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56997.log.gz 2026-02-17T05:18:21.389 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95619.log 2026-02-17T05:18:21.390 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57830.log 2026-02-17T05:18:21.390 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108459.log.gz 2026-02-17T05:18:21.390 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95619.log.gz 2026-02-17T05:18:21.390 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83741.log 2026-02-17T05:18:21.390 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92971.log 2026-02-17T05:18:21.390 INFO:teuthology.orchestra.run.trial100.stderr: 29.9% -- replaced with /var/log/ceph/ceph-client.admin.57830.log.gz 2026-02-17T05:18:21.390 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83741.log.gz 2026-02-17T05:18:21.390 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112036.log 2026-02-17T05:18:21.391 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79330.log 2026-02-17T05:18:21.391 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92971.log.gz 2026-02-17T05:18:21.391 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112036.log.gz 2026-02-17T05:18:21.391 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81633.log 2026-02-17T05:18:21.391 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79330.log: 29.2% -- replaced with /var/log/ceph/ceph-client.admin.79330.log.gz 2026-02-17T05:18:21.391 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115236.log 2026-02-17T05:18:21.391 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81633.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.81633.log.gz -- 2026-02-17T05:18:21.392 INFO:teuthology.orchestra.run.trial100.stderr: /var/log/ceph/ceph-client.admin.63579.log 2026-02-17T05:18:21.392 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.115236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115236.log.gz 2026-02-17T05:18:21.392 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82515.log 2026-02-17T05:18:21.392 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63579.log.gz 2026-02-17T05:18:21.392 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55282.log 2026-02-17T05:18:21.392 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82515.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.59889.log -- replaced with /var/log/ceph/ceph-client.admin.82515.log.gz 2026-02-17T05:18:21.392 INFO:teuthology.orchestra.run.trial100.stderr: 2026-02-17T05:18:21.392 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55282.log.gz 2026-02-17T05:18:21.393 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88397.log 2026-02-17T05:18:21.393 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59889.log.gz 2026-02-17T05:18:21.393 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50635.log 2026-02-17T05:18:21.393 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88397.log.gz 2026-02-17T05:18:21.393 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107821.log 2026-02-17T05:18:21.393 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100226.log 2026-02-17T05:18:21.394 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50635.log.gz 2026-02-17T05:18:21.394 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107821.log.gz 2026-02-17T05:18:21.394 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95031.log 2026-02-17T05:18:21.394 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114011.log 2026-02-17T05:18:21.394 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100226.log.gz 2026-02-17T05:18:21.394 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95031.log.gz 2026-02-17T05:18:21.394 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76929.log 2026-02-17T05:18:21.394 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97432.log 2026-02-17T05:18:21.394 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114011.log.gz 2026-02-17T05:18:21.395 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76929.log.gz 2026-02-17T05:18:21.395 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49980.log 2026-02-17T05:18:21.395 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72617.log 2026-02-17T05:18:21.395 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97432.log.gz 2026-02-17T05:18:21.395 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49980.log.gz 2026-02-17T05:18:21.395 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93267.log 2026-02-17T05:18:21.395 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63040.log 2026-02-17T05:18:21.395 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72617.log.gz 2026-02-17T05:18:21.396 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93267.log.gz 2026-02-17T05:18:21.396 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101892.log 2026-02-17T05:18:21.396 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104048.log 2026-02-17T05:18:21.396 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63040.log.gz 2026-02-17T05:18:21.396 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101892.log.gz 2026-02-17T05:18:21.396 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63236.log 2026-02-17T05:18:21.396 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71143.log 2026-02-17T05:18:21.397 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104048.log.gz 2026-02-17T05:18:21.397 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63236.log.gz 2026-02-17T05:18:21.397 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87073.log 2026-02-17T05:18:21.397 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74332.log 2026-02-17T05:18:21.397 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71143.log.gz 2026-02-17T05:18:21.397 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87073.log.gz 2026-02-17T05:18:21.397 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100373.log 2026-02-17T05:18:21.397 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64805.log 2026-02-17T05:18:21.398 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74332.log.gz 2026-02-17T05:18:21.398 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100373.log.gz 2026-02-17T05:18:21.398 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80163.log 2026-02-17T05:18:21.398 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94198.log 2026-02-17T05:18:21.398 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64805.log.gz 2026-02-17T05:18:21.398 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80163.log.gz 2026-02-17T05:18:21.398 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65148.log 2026-02-17T05:18:21.398 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63089.log 2026-02-17T05:18:21.399 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94198.log.gz 2026-02-17T05:18:21.399 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65148.log.gz 2026-02-17T05:18:21.399 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83251.log 2026-02-17T05:18:21.399 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88691.log 2026-02-17T05:18:21.399 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63089.log.gz 2026-02-17T05:18:21.399 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83251.log.gz 2026-02-17T05:18:21.399 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68140.log 2026-02-17T05:18:21.399 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66716.log 2026-02-17T05:18:21.400 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88691.log.gz 2026-02-17T05:18:21.400 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68140.log.gz 2026-02-17T05:18:21.400 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68091.log 2026-02-17T05:18:21.400 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108998.log 2026-02-17T05:18:21.400 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66716.log.gz 2026-02-17T05:18:21.400 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68091.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68091.log.gz 2026-02-17T05:18:21.400 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75704.log 2026-02-17T05:18:21.401 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59938.log 2026-02-17T05:18:21.401 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108998.log.gz 2026-02-17T05:18:21.401 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75704.log.gz 2026-02-17T05:18:21.401 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86093.log 2026-02-17T05:18:21.401 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59938.log.gz 2026-02-17T05:18:21.401 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111448.log 2026-02-17T05:18:21.401 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86093.log.gz 2026-02-17T05:18:21.401 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83005.log 2026-02-17T05:18:21.402 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111448.log.gz 2026-02-17T05:18:21.402 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109439.log 2026-02-17T05:18:21.402 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83005.log.gz 2026-02-17T05:18:21.402 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90410.log 2026-02-17T05:18:21.402 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44083.log 2026-02-17T05:18:21.402 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109439.log.gz 2026-02-17T05:18:21.402 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90410.log.gz 2026-02-17T05:18:21.402 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101500.log 2026-02-17T05:18:21.403 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.44083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44083.log.gz 2026-02-17T05:18:21.403 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91991.log 2026-02-17T05:18:21.403 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101500.log.gz 2026-02-17T05:18:21.403 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90065.log 2026-02-17T05:18:21.403 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91991.log.gz 2026-02-17T05:18:21.403 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66520.log 2026-02-17T05:18:21.403 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90065.log.gz 2026-02-17T05:18:21.403 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104293.log 2026-02-17T05:18:21.404 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66520.log.gz 2026-02-17T05:18:21.404 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73058.log 2026-02-17T05:18:21.404 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104293.log.gz 2026-02-17T05:18:21.404 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82319.log 2026-02-17T05:18:21.404 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73058.log.gz 2026-02-17T05:18:21.404 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90312.log 2026-02-17T05:18:21.404 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82319.log.gz 2026-02-17T05:18:21.404 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106498.log 2026-02-17T05:18:21.405 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90312.log.gz 2026-02-17T05:18:21.405 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85456.log 2026-02-17T05:18:21.405 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87220.log 2026-02-17T05:18:21.405 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106498.log.gz 2026-02-17T05:18:21.405 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85456.log.gz 2026-02-17T05:18:21.405 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85407.log 2026-02-17T05:18:21.405 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51132.log 2026-02-17T05:18:21.406 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87220.log.gz 2026-02-17T05:18:21.406 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85407.log.gz 2026-02-17T05:18:21.406 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49556.log 2026-02-17T05:18:21.406 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51132.log.gz 2026-02-17T05:18:21.406 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100520.log 2026-02-17T05:18:21.406 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49556.log.gz 2026-02-17T05:18:21.406 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80457.log 2026-02-17T05:18:21.406 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100520.log.gz 2026-02-17T05:18:21.406 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73254.log 2026-02-17T05:18:21.407 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80457.log.gz 2026-02-17T05:18:21.407 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111987.log 2026-02-17T05:18:21.407 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73254.log.gz 2026-02-17T05:18:21.407 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59693.log 2026-02-17T05:18:21.407 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111987.log.gz 2026-02-17T05:18:21.407 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102872.log 2026-02-17T05:18:21.407 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59693.log.gz 2026-02-17T05:18:21.408 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97580.log 2026-02-17T05:18:21.408 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102872.log.gz 2026-02-17T05:18:21.408 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88054.log 2026-02-17T05:18:21.408 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97580.log.gz 2026-02-17T05:18:21.408 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102774.log 2026-02-17T05:18:21.408 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88054.log.gz 2026-02-17T05:18:21.408 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112637.log 2026-02-17T05:18:21.408 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102774.log.gz 2026-02-17T05:18:21.408 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84329.log 2026-02-17T05:18:21.409 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112637.log.gz 2026-02-17T05:18:21.409 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98560.log 2026-02-17T05:18:21.409 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84329.log.gz 2026-02-17T05:18:21.409 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79575.log 2026-02-17T05:18:21.409 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98560.log.gz 2026-02-17T05:18:21.409 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53763.log 2026-02-17T05:18:21.409 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79575.log.gz 2026-02-17T05:18:21.409 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72666.log 2026-02-17T05:18:21.410 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53763.log.gz 2026-02-17T05:18:21.410 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98217.log 2026-02-17T05:18:21.410 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72666.log.gz 2026-02-17T05:18:21.410 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111399.log 2026-02-17T05:18:21.410 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98217.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98217.log.gz 2026-02-17T05:18:21.410 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53509.log 2026-02-17T05:18:21.410 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111399.log.gz 2026-02-17T05:18:21.410 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66765.log 2026-02-17T05:18:21.411 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75361.log 2026-02-17T05:18:21.411 INFO:teuthology.orchestra.run.trial100.stderr: 32.1% -- replaced with /var/log/ceph/ceph-client.admin.53509.log.gz 2026-02-17T05:18:21.411 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66765.log.gz 2026-02-17T05:18:21.411 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101696.log 2026-02-17T05:18:21.411 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106890.log 2026-02-17T05:18:21.411 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75361.log.gz 2026-02-17T05:18:21.411 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101696.log.gz 2026-02-17T05:18:21.411 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70456.log 2026-02-17T05:18:21.412 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106890.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-17T05:18:21.412 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106890.log.gz 2026-02-17T05:18:21.412 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70456.log.gz 2026-02-17T05:18:21.412 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96305.log 2026-02-17T05:18:21.412 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80212.log 2026-02-17T05:18:21.412 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96305.log.gz 2026-02-17T05:18:21.412 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49785.log 2026-02-17T05:18:21.413 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80212.log.gz 2026-02-17T05:18:21.413 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58222.log 2026-02-17T05:18:21.413 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49785.log.gz 2026-02-17T05:18:21.413 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65295.log 2026-02-17T05:18:21.413 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58222.log.gz 2026-02-17T05:18:21.413 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64020.log 2026-02-17T05:18:21.413 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65295.log.gz 2026-02-17T05:18:21.413 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77076.log 2026-02-17T05:18:21.414 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64020.log.gz 2026-02-17T05:18:21.414 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84084.log 2026-02-17T05:18:21.414 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77076.log.gz 2026-02-17T05:18:21.414 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75018.log 2026-02-17T05:18:21.414 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84084.log.gz 2026-02-17T05:18:21.414 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62290.log 2026-02-17T05:18:21.414 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75018.log.gz 2026-02-17T05:18:21.414 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106106.log 2026-02-17T05:18:21.415 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62290.log.gz 2026-02-17T05:18:21.415 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93904.log 2026-02-17T05:18:21.415 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106106.log.gz 2026-02-17T05:18:21.415 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104440.log 2026-02-17T05:18:21.415 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93904.log.gz 2026-02-17T05:18:21.415 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98952.log 2026-02-17T05:18:21.415 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104440.log.gz 2026-02-17T05:18:21.415 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88887.log 2026-02-17T05:18:21.416 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98952.log.gz 2026-02-17T05:18:21.416 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105273.log 2026-02-17T05:18:21.416 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88887.log.gz 2026-02-17T05:18:21.416 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81731.log 2026-02-17T05:18:21.416 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105273.log.gz 2026-02-17T05:18:21.416 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82172.log 2026-02-17T05:18:21.416 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81731.log.gz 2026-02-17T05:18:21.416 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100471.log 2026-02-17T05:18:21.417 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82172.log.gz 2026-02-17T05:18:21.417 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79771.log 2026-02-17T05:18:21.417 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100471.log.gz 2026-02-17T05:18:21.417 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52910.log 2026-02-17T05:18:21.417 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79771.log.gz 2026-02-17T05:18:21.417 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96452.log 2026-02-17T05:18:21.417 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.52910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52910.log.gz 2026-02-17T05:18:21.417 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60428.log 2026-02-17T05:18:21.418 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96452.log.gz 2026-02-17T05:18:21.418 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97629.log 2026-02-17T05:18:21.418 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60428.log.gz 2026-02-17T05:18:21.418 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60918.log 2026-02-17T05:18:21.418 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97629.log.gz 2026-02-17T05:18:21.418 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59742.log 2026-02-17T05:18:21.418 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60918.log.gz 2026-02-17T05:18:21.418 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73989.log 2026-02-17T05:18:21.419 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59742.log.gz 2026-02-17T05:18:21.419 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62501.log 2026-02-17T05:18:21.419 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73989.log.gz 2026-02-17T05:18:21.419 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69953.log 2026-02-17T05:18:21.419 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62501.log.gz 2026-02-17T05:18:21.419 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58663.log 2026-02-17T05:18:21.419 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69953.log.gz 2026-02-17T05:18:21.419 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48900.log 2026-02-17T05:18:21.419 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58663.log.gz 2026-02-17T05:18:21.420 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105371.log 2026-02-17T05:18:21.420 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48900.log.gz 2026-02-17T05:18:21.420 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78105.log 2026-02-17T05:18:21.420 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105371.log.gz 2026-02-17T05:18:21.420 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44786.log 2026-02-17T05:18:21.420 INFO:teuthology.orchestra.run.trial100.stderr: 95.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-17T05:18:21.420 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78105.log.gz 2026-02-17T05:18:21.420 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110321.log 2026-02-17T05:18:21.420 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.44786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44786.log.gz 2026-02-17T05:18:21.421 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77027.log 2026-02-17T05:18:21.421 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110321.log.gz 2026-02-17T05:18:21.421 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86142.log 2026-02-17T05:18:21.421 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77027.log.gz 2026-02-17T05:18:21.421 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60085.log 2026-02-17T05:18:21.421 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86142.log.gz 2026-02-17T05:18:21.421 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80604.log 2026-02-17T05:18:21.421 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60085.log.gz 2026-02-17T05:18:21.421 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102529.log 2026-02-17T05:18:21.422 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80604.log.gz 2026-02-17T05:18:21.422 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98315.log 2026-02-17T05:18:21.422 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102529.log.gz 2026-02-17T05:18:21.422 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2026-02-17T05:18:21.422 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98315.log.gz 2026-02-17T05:18:21.422 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58418.log 2026-02-17T05:18:21.422 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58271.log 2026-02-17T05:18:21.423 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58418.log.gz 2026-02-17T05:18:21.423 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66667.log 2026-02-17T05:18:21.423 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58271.log.gz 2026-02-17T05:18:21.423 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109145.log 2026-02-17T05:18:21.423 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59791.log 2026-02-17T05:18:21.423 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66667.log.gz 2026-02-17T05:18:21.424 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71094.log 2026-02-17T05:18:21.424 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109145.log.gz 2026-02-17T05:18:21.424 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59791.log.gz 2026-02-17T05:18:21.424 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77566.log 2026-02-17T05:18:21.424 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90851.log 2026-02-17T05:18:21.424 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71094.log.gz 2026-02-17T05:18:21.424 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77566.log.gz 2026-02-17T05:18:21.424 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74969.log 2026-02-17T05:18:21.425 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90851.log.gz 2026-02-17T05:18:21.425 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75312.log 2026-02-17T05:18:21.425 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74969.log.gz 2026-02-17T05:18:21.425 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98658.log 2026-02-17T05:18:21.425 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112490.log 2026-02-17T05:18:21.425 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75312.log.gz 2026-02-17T05:18:21.425 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98658.log.gz 2026-02-17T05:18:21.426 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75900.log 2026-02-17T05:18:21.426 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62192.log 2026-02-17T05:18:21.426 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112490.log.gz 2026-02-17T05:18:21.426 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75900.log.gz 2026-02-17T05:18:21.426 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73597.log 2026-02-17T05:18:21.426 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71045.log 2026-02-17T05:18:21.426 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62192.log.gz 2026-02-17T05:18:21.427 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73597.log.gz 2026-02-17T05:18:21.427 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85309.log 2026-02-17T05:18:21.427 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103166.log 2026-02-17T05:18:21.427 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71045.log.gz 2026-02-17T05:18:21.427 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85309.log.gz 2026-02-17T05:18:21.427 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109096.log 2026-02-17T05:18:21.427 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103166.log.gz 2026-02-17T05:18:21.427 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85113.log 2026-02-17T05:18:21.428 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67503.log 2026-02-17T05:18:21.428 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109096.log.gz 2026-02-17T05:18:21.428 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85113.log.gz 2026-02-17T05:18:21.428 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48704.log 2026-02-17T05:18:21.428 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67503.log.gz 2026-02-17T05:18:21.428 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71241.log 2026-02-17T05:18:21.428 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48704.log.gz 2026-02-17T05:18:21.429 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111497.log 2026-02-17T05:18:21.429 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71241.log.gz 2026-02-17T05:18:21.429 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70149.log 2026-02-17T05:18:21.429 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111497.log.gz 2026-02-17T05:18:21.429 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88544.log 2026-02-17T05:18:21.429 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70149.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.70149.log.gz 2026-02-17T05:18:21.429 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59153.log 2026-02-17T05:18:21.430 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72813.log 2026-02-17T05:18:21.430 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88544.log.gz 2026-02-17T05:18:21.430 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67601.log 2026-02-17T05:18:21.430 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59153.log.gz 2026-02-17T05:18:21.430 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72813.log.gz 2026-02-17T05:18:21.430 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45016.log 2026-02-17T05:18:21.430 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83643.log 2026-02-17T05:18:21.431 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67601.log.gz 2026-02-17T05:18:21.431 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.45016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45016.log.gz 2026-02-17T05:18:21.431 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70848.log 2026-02-17T05:18:21.431 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83643.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83643.log.gz 2026-02-17T05:18:21.431 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100765.log 2026-02-17T05:18:21.431 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70848.log.gz 2026-02-17T05:18:21.431 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77860.log 2026-02-17T05:18:21.431 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100765.log.gz 2026-02-17T05:18:21.432 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109635.log 2026-02-17T05:18:21.432 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77860.log.gz 2026-02-17T05:18:21.432 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102921.log 2026-02-17T05:18:21.432 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109635.log.gz 2026-02-17T05:18:21.432 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64609.log 2026-02-17T05:18:21.432 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102921.log.gz 2026-02-17T05:18:21.432 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91243.log 2026-02-17T05:18:21.433 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64609.log.gz 2026-02-17T05:18:21.433 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74381.log 2026-02-17T05:18:21.433 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91243.log.gz 2026-02-17T05:18:21.433 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112980.log 2026-02-17T05:18:21.433 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74381.log.gz 2026-02-17T05:18:21.433 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107135.log 2026-02-17T05:18:21.433 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112980.log.gz 2026-02-17T05:18:21.433 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90263.log 2026-02-17T05:18:21.433 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107135.log.gz 2026-02-17T05:18:21.434 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100030.log 2026-02-17T05:18:21.434 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90263.log.gz 2026-02-17T05:18:21.434 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68042.log 2026-02-17T05:18:21.434 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100030.log.gz 2026-02-17T05:18:21.434 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112379.log 2026-02-17T05:18:21.434 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68042.log.gz 2026-02-17T05:18:21.434 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113178.log 2026-02-17T05:18:21.434 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112379.log.gz 2026-02-17T05:18:21.435 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87122.log 2026-02-17T05:18:21.435 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113178.log.gz 2026-02-17T05:18:21.435 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-17T05:18:21.435 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87122.log.gz 2026-02-17T05:18:21.435 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72421.log 2026-02-17T05:18:21.435 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57242.log 2026-02-17T05:18:21.436 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72421.log: 30.0% -- replaced with /var/log/ceph/ceph-client.admin.72421.log.gz 2026-02-17T05:18:21.436 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89721.log 2026-02-17T05:18:21.436 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57242.log.gz 2026-02-17T05:18:21.436 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89084.log 2026-02-17T05:18:21.436 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89721.log.gz 2026-02-17T05:18:21.436 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56899.log 2026-02-17T05:18:21.436 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89084.log.gz 2026-02-17T05:18:21.436 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114501.log 2026-02-17T05:18:21.437 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56899.log.gz 2026-02-17T05:18:21.437 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92334.log 2026-02-17T05:18:21.437 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114501.log.gz 2026-02-17T05:18:21.437 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108410.log 2026-02-17T05:18:21.437 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92334.log.gz 2026-02-17T05:18:21.437 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59399.log 2026-02-17T05:18:21.437 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85652.log 2026-02-17T05:18:21.437 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108410.log.gz 2026-02-17T05:18:21.438 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59399.log.gz 2026-02-17T05:18:21.438 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76145.log 2026-02-17T05:18:21.438 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85652.log.gz 2026-02-17T05:18:21.438 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56164.log 2026-02-17T05:18:21.438 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60330.log 2026-02-17T05:18:21.438 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76145.log.gz 2026-02-17T05:18:21.438 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56164.log.gz 2026-02-17T05:18:21.438 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93118.log 2026-02-17T05:18:21.439 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60330.log.gz 2026-02-17T05:18:21.439 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103362.log 2026-02-17T05:18:21.439 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48899.log 2026-02-17T05:18:21.439 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93118.log.gz 2026-02-17T05:18:21.439 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103362.log.gz 2026-02-17T05:18:21.439 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97678.log 2026-02-17T05:18:21.439 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48899.log.gz 2026-02-17T05:18:21.439 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112134.log 2026-02-17T05:18:21.440 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97678.log.gz 2026-02-17T05:18:21.440 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101549.log 2026-02-17T05:18:21.440 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114060.log 2026-02-17T05:18:21.440 INFO:teuthology.orchestra.run.trial100.stderr: 30.5% -- replaced with /var/log/ceph/ceph-client.admin.112134.log.gz 2026-02-17T05:18:21.440 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101549.log.gz 2026-02-17T05:18:21.440 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93708.log 2026-02-17T05:18:21.440 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114060.log.gz 2026-02-17T05:18:21.440 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68924.log 2026-02-17T05:18:21.441 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93708.log.gz 2026-02-17T05:18:21.441 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68385.log 2026-02-17T05:18:21.441 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68924.log.gz 2026-02-17T05:18:21.441 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96893.log 2026-02-17T05:18:21.441 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68385.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68385.log.gz 2026-02-17T05:18:21.441 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48355.log 2026-02-17T05:18:21.441 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96893.log.gz 2026-02-17T05:18:21.441 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60869.log 2026-02-17T05:18:21.441 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48355.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48355.log.gz 2026-02-17T05:18:21.442 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96942.log 2026-02-17T05:18:21.442 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60869.log.gz 2026-02-17T05:18:21.442 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72078.log 2026-02-17T05:18:21.442 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96942.log.gz 2026-02-17T05:18:21.442 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92628.log 2026-02-17T05:18:21.442 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72078.log.gz 2026-02-17T05:18:21.442 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62697.log 2026-02-17T05:18:21.442 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92628.log.gz 2026-02-17T05:18:21.443 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77615.log 2026-02-17T05:18:21.443 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62697.log.gz 2026-02-17T05:18:21.443 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69414.log 2026-02-17T05:18:21.443 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77615.log.gz 2026-02-17T05:18:21.443 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99050.log 2026-02-17T05:18:21.443 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69414.log.gz 2026-02-17T05:18:21.443 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111791.log 2026-02-17T05:18:21.443 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99050.log.gz 2026-02-17T05:18:21.443 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53080.log 2026-02-17T05:18:21.444 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111791.log.gz 2026-02-17T05:18:21.444 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77125.log 2026-02-17T05:18:21.444 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53080.log.gz 2026-02-17T05:18:21.444 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95227.log 2026-02-17T05:18:21.444 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77125.log.gz 2026-02-17T05:18:21.444 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97531.log 2026-02-17T05:18:21.444 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95227.log.gz 2026-02-17T05:18:21.444 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107478.log 2026-02-17T05:18:21.445 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97531.log.gz 2026-02-17T05:18:21.445 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102235.log 2026-02-17T05:18:21.445 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107478.log.gz 2026-02-17T05:18:21.445 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73793.log 2026-02-17T05:18:21.445 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102235.log.gz 2026-02-17T05:18:21.445 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56948.log 2026-02-17T05:18:21.445 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73793.log.gz 2026-02-17T05:18:21.445 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92383.log 2026-02-17T05:18:21.446 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56948.log.gz 2026-02-17T05:18:21.446 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95080.log 2026-02-17T05:18:21.446 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88789.log 2026-02-17T05:18:21.446 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92383.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92383.log.gz 2026-02-17T05:18:21.446 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95080.log.gz 2026-02-17T05:18:21.446 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81388.log 2026-02-17T05:18:21.446 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88789.log.gz 2026-02-17T05:18:21.446 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105665.log 2026-02-17T05:18:21.447 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81388.log.gz 2026-02-17T05:18:21.447 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73401.log 2026-02-17T05:18:21.447 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105665.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.64462.log -- replaced with /var/log/ceph/ceph-client.admin.105665.log.gz 2026-02-17T05:18:21.447 INFO:teuthology.orchestra.run.trial100.stderr: 2026-02-17T05:18:21.447 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73401.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.73401.log.gz 2026-02-17T05:18:21.447 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.106596.log 2026-02-17T05:18:21.447 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64462.log.gz 2026-02-17T05:18:21.448 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75116.log 2026-02-17T05:18:21.448 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106596.log.gz 2026-02-17T05:18:21.448 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93757.log 2026-02-17T05:18:21.448 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75116.log.gz 2026-02-17T05:18:21.448 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95864.log 2026-02-17T05:18:21.448 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93757.log.gz 2026-02-17T05:18:21.448 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77468.log 2026-02-17T05:18:21.448 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95864.log.gz 2026-02-17T05:18:21.448 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112735.log 2026-02-17T05:18:21.449 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77468.log.gz 2026-02-17T05:18:21.449 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70198.log 2026-02-17T05:18:21.449 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112735.log.gz 2026-02-17T05:18:21.449 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62746.log 2026-02-17T05:18:21.449 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70198.log.gz 2026-02-17T05:18:21.449 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104783.log 2026-02-17T05:18:21.449 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62746.log.gz 2026-02-17T05:18:21.450 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109684.log 2026-02-17T05:18:21.450 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104783.log.gz 2026-02-17T05:18:21.450 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56017.log 2026-02-17T05:18:21.450 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109684.log.gz 2026-02-17T05:18:21.450 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112833.log 2026-02-17T05:18:21.450 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70946.log 2026-02-17T05:18:21.450 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56017.log.gz 2026-02-17T05:18:21.451 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112833.log.gz 2026-02-17T05:18:21.451 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113031.log 2026-02-17T05:18:21.451 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70946.log.gz 2026-02-17T05:18:21.451 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84427.log 2026-02-17T05:18:21.451 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113031.log.gz 2026-02-17T05:18:21.451 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92873.log 2026-02-17T05:18:21.451 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84427.log.gz 2026-02-17T05:18:21.451 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72715.log 2026-02-17T05:18:21.452 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92873.log.gz 2026-02-17T05:18:21.452 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61604.log 2026-02-17T05:18:21.452 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72715.log.gz 2026-02-17T05:18:21.452 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95178.log 2026-02-17T05:18:21.452 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61604.log.gz 2026-02-17T05:18:21.452 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115138.log 2026-02-17T05:18:21.452 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95178.log.gz 2026-02-17T05:18:21.453 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114550.log 2026-02-17T05:18:21.453 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.115138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115138.log.gz 2026-02-17T05:18:21.453 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111105.log 2026-02-17T05:18:21.453 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114550.log.gz 2026-02-17T05:18:21.453 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115285.log 2026-02-17T05:18:21.453 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111105.log.gz 2026-02-17T05:18:21.453 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97825.log 2026-02-17T05:18:21.453 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.115285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115285.log.gz 2026-02-17T05:18:21.454 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54057.log 2026-02-17T05:18:21.454 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97825.log.gz 2026-02-17T05:18:21.454 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112539.log 2026-02-17T05:18:21.454 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54057.log.gz 2026-02-17T05:18:21.454 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96795.log 2026-02-17T05:18:21.454 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112539.log.gz 2026-02-17T05:18:21.454 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79183.log 2026-02-17T05:18:21.455 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54743.log 2026-02-17T05:18:21.455 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96795.log.gz 2026-02-17T05:18:21.455 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79183.log.gz 2026-02-17T05:18:21.455 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95668.log 2026-02-17T05:18:21.455 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54743.log.gz 2026-02-17T05:18:21.455 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70100.log 2026-02-17T05:18:21.455 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95668.log.gz 2026-02-17T05:18:21.456 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81290.log 2026-02-17T05:18:21.456 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70100.log.gz 2026-02-17T05:18:21.456 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74871.log 2026-02-17T05:18:21.456 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81290.log.gz 2026-02-17T05:18:21.456 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49456.log 2026-02-17T05:18:21.456 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74871.log.gz 2026-02-17T05:18:21.456 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102823.log 2026-02-17T05:18:21.456 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49456.log.gz 2026-02-17T05:18:21.457 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63824.log 2026-02-17T05:18:21.457 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102823.log.gz 2026-02-17T05:18:21.457 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63481.log 2026-02-17T05:18:21.457 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63824.log.gz 2026-02-17T05:18:21.457 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61800.log 2026-02-17T05:18:21.457 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63481.log.gz 2026-02-17T05:18:21.457 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62339.log 2026-02-17T05:18:21.457 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61800.log.gz 2026-02-17T05:18:21.458 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58614.log 2026-02-17T05:18:21.458 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62339.log.gz 2026-02-17T05:18:21.458 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79232.log 2026-02-17T05:18:21.458 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58614.log.gz 2026-02-17T05:18:21.458 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105469.log 2026-02-17T05:18:21.458 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79232.log.gz 2026-02-17T05:18:21.458 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71731.log 2026-02-17T05:18:21.458 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105469.log.gz 2026-02-17T05:18:21.459 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77272.log 2026-02-17T05:18:21.459 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71731.log.gz 2026-02-17T05:18:21.459 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60820.log 2026-02-17T05:18:21.459 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77272.log.gz 2026-02-17T05:18:21.459 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103509.log 2026-02-17T05:18:21.459 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60820.log.gz 2026-02-17T05:18:21.459 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103411.log 2026-02-17T05:18:21.460 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103509.log.gz 2026-02-17T05:18:21.460 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102284.log 2026-02-17T05:18:21.460 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103411.log.gz 2026-02-17T05:18:21.460 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66618.log 2026-02-17T05:18:21.460 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102284.log.gz 2026-02-17T05:18:21.460 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71682.log 2026-02-17T05:18:21.460 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66618.log.gz 2026-02-17T05:18:21.460 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101108.log 2026-02-17T05:18:21.461 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71682.log.gz 2026-02-17T05:18:21.461 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81829.log 2026-02-17T05:18:21.461 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101108.log.gz 2026-02-17T05:18:21.461 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63285.log 2026-02-17T05:18:21.461 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81829.log.gz 2026-02-17T05:18:21.461 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100324.log 2026-02-17T05:18:21.461 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63285.log.gz 2026-02-17T05:18:21.461 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105812.log 2026-02-17T05:18:21.462 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100324.log.gz 2026-02-17T05:18:21.462 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106792.log 2026-02-17T05:18:21.462 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105812.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105812.log.gz 2026-02-17T05:18:21.462 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90949.log 2026-02-17T05:18:21.462 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71584.log 2026-02-17T05:18:21.462 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106792.log.gz 2026-02-17T05:18:21.463 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90949.log.gz 2026-02-17T05:18:21.463 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51279.log 2026-02-17T05:18:21.463 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.71584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83202.log 2026-02-17T05:18:21.463 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71584.log.gz 2026-02-17T05:18:21.463 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59350.log 2026-02-17T05:18:21.463 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51279.log.gz 2026-02-17T05:18:21.463 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83202.log.gz 2026-02-17T05:18:21.463 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108900.log 2026-02-17T05:18:21.463 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60477.log 2026-02-17T05:18:21.464 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59350.log.gz 2026-02-17T05:18:21.464 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108900.log.gz 2026-02-17T05:18:21.464 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82907.log 2026-02-17T05:18:21.464 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60477.log.gz 2026-02-17T05:18:21.464 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70750.log 2026-02-17T05:18:21.464 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69267.log 2026-02-17T05:18:21.464 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82907.log.gz 2026-02-17T05:18:21.464 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70750.log.gz 2026-02-17T05:18:21.464 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107968.log 2026-02-17T05:18:21.465 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69267.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.69267.log.gz --verbose 2026-02-17T05:18:21.465 INFO:teuthology.orchestra.run.trial100.stderr: -- /var/log/ceph/ceph-client.admin.67405.log 2026-02-17T05:18:21.465 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107968.log.gz 2026-02-17T05:18:21.465 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63775.log 2026-02-17T05:18:21.465 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67405.log.gz 2026-02-17T05:18:21.465 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79722.log 2026-02-17T05:18:21.465 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106302.log 2026-02-17T05:18:21.465 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63775.log.gz 2026-02-17T05:18:21.466 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79722.log.gz 2026-02-17T05:18:21.466 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113472.log 2026-02-17T05:18:21.466 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100275.log 2026-02-17T05:18:21.466 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106302.log.gz 2026-02-17T05:18:21.466 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113570.log 2026-02-17T05:18:21.466 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113472.log.gz 2026-02-17T05:18:21.466 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96991.log 2026-02-17T05:18:21.466 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100275.log.gz 2026-02-17T05:18:21.466 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107772.log 2026-02-17T05:18:21.466 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113570.log.gz 2026-02-17T05:18:21.467 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111938.log 2026-02-17T05:18:21.467 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96991.log.gz 2026-02-17T05:18:21.467 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84819.log 2026-02-17T05:18:21.467 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107772.log.gz 2026-02-17T05:18:21.467 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85848.log 2026-02-17T05:18:21.467 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111938.log.gz 2026-02-17T05:18:21.467 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86387.log 2026-02-17T05:18:21.467 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84819.log.gz 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104636.log 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85848.log.gz 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86828.log 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86387.log.gz 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89427.log 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104636.log.gz 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63187.log 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86828.log.gz 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89427.log.gzgzip 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.75459.log 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93953.log 2026-02-17T05:18:21.468 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63187.log.gz 2026-02-17T05:18:21.469 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64658.log 2026-02-17T05:18:21.469 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75459.log.gz 2026-02-17T05:18:21.469 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52783.log 2026-02-17T05:18:21.469 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93953.log.gz 2026-02-17T05:18:21.469 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64658.log.gz 2026-02-17T05:18:21.469 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70345.log 2026-02-17T05:18:21.469 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.52783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83986.log 2026-02-17T05:18:21.469 INFO:teuthology.orchestra.run.trial100.stderr: 31.5% -- replaced with /var/log/ceph/ceph-client.admin.52783.log.gz 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115089.log 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70345.log.gz 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59497.log 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83986.log.gz 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.115089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59448.log 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115089.log.gz 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59497.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.86534.log -- replaced with /var/log/ceph/ceph-client.admin.59497.log.gz 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr: 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86877.log 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59448.log.gz 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82466.log 2026-02-17T05:18:21.470 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86534.log.gz 2026-02-17T05:18:21.471 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103558.log 2026-02-17T05:18:21.471 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86877.log.gz 2026-02-17T05:18:21.471 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110223.log 2026-02-17T05:18:21.471 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82466.log.gz 2026-02-17T05:18:21.471 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64707.log 2026-02-17T05:18:21.471 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103558.log.gz 2026-02-17T05:18:21.471 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50319.log 2026-02-17T05:18:21.471 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110223.log.gz 2026-02-17T05:18:21.471 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61506.log 2026-02-17T05:18:21.471 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64707.log.gz 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98511.log 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50319.log.gz 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75606.log 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61506.log.gz 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89280.log 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98511.log.gz 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58761.log 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75606.log.gz 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63677.log 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89280.log.gz 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56066.log 2026-02-17T05:18:21.472 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58761.log.gz 2026-02-17T05:18:21.473 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80898.log 2026-02-17T05:18:21.473 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63677.log.gz 2026-02-17T05:18:21.473 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108606.log 2026-02-17T05:18:21.473 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56066.log.gz 2026-02-17T05:18:21.473 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86975.log 2026-02-17T05:18:21.473 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80898.log.gz 2026-02-17T05:18:21.473 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53082.log 2026-02-17T05:18:21.473 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108606.log.gz 2026-02-17T05:18:21.473 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66128.log 2026-02-17T05:18:21.473 INFO:teuthology.orchestra.run.trial100.stderr: 31.5% -- replaced with /var/log/ceph/ceph-client.admin.86975.log.gz 2026-02-17T05:18:21.474 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58810.log 2026-02-17T05:18:21.474 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53082.log.gz 2026-02-17T05:18:21.474 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104685.log 2026-02-17T05:18:21.474 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66128.log.gz 2026-02-17T05:18:21.474 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66030.log 2026-02-17T05:18:21.474 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58810.log.gz 2026-02-17T05:18:21.474 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72764.log 2026-02-17T05:18:21.474 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104685.log.gz 2026-02-17T05:18:21.474 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55086.log 2026-02-17T05:18:21.474 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66030.log.gz 2026-02-17T05:18:21.475 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68238.log 2026-02-17T05:18:21.475 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72764.log.gz 2026-02-17T05:18:21.475 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67797.log 2026-02-17T05:18:21.475 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55086.log.gz 2026-02-17T05:18:21.475 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49981.log 2026-02-17T05:18:21.475 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68238.log.gz 2026-02-17T05:18:21.475 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53361.log 2026-02-17T05:18:21.475 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67797.log.gz 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110615.log 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49981.log.gz 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101255.log 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53361.log.gz 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104979.log 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110615.log.gz 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101255.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.101255.log.gz 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.96403.log 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95374.log 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104979.log.gz 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53665.log 2026-02-17T05:18:21.476 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96403.log.gz 2026-02-17T05:18:21.477 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95374.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.95374.log.gz 2026-02-17T05:18:21.477 INFO:teuthology.orchestra.run.trial100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51083.log 2026-02-17T05:18:21.477 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95913.log 2026-02-17T05:18:21.477 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53665.log.gz 2026-02-17T05:18:21.477 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99736.log 2026-02-17T05:18:21.477 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51083.log.gz 2026-02-17T05:18:21.477 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95913.log.gz 2026-02-17T05:18:21.477 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110370.log 2026-02-17T05:18:21.477 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74234.log 2026-02-17T05:18:21.477 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99736.log.gz 2026-02-17T05:18:21.478 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111693.log 2026-02-17T05:18:21.478 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110370.log.gz 2026-02-17T05:18:21.478 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74234.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.74234.log.gz -5 2026-02-17T05:18:21.478 INFO:teuthology.orchestra.run.trial100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.80800.log 2026-02-17T05:18:21.478 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102725.log 2026-02-17T05:18:21.478 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111693.log.gz 2026-02-17T05:18:21.478 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115040.log 2026-02-17T05:18:21.478 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80800.log.gz 2026-02-17T05:18:21.478 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88740.log 2026-02-17T05:18:21.478 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102725.log.gz 2026-02-17T05:18:21.479 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.115040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115040.log.gz 2026-02-17T05:18:21.479 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87564.log 2026-02-17T05:18:21.479 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103264.log 2026-02-17T05:18:21.479 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88740.log.gz 2026-02-17T05:18:21.479 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69512.log 2026-02-17T05:18:21.479 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87564.log.gz 2026-02-17T05:18:21.479 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111007.log 2026-02-17T05:18:21.479 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103264.log.gz 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55772.log 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69512.log.gz 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50123.log 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111007.log.gz 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90655.log 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55772.log.gz 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95521.log 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50123.log.gz 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96256.log 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90655.log.gz 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66177.log 2026-02-17T05:18:21.480 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95521.log.gz 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100569.log 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96256.log.gz 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75998.log 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66177.log.gz 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102627.log 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100569.log.gz 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93806.log 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75998.log.gz 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67160.log 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102627.log.gz 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111056.log 2026-02-17T05:18:21.481 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93806.log.gz 2026-02-17T05:18:21.482 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53278.log 2026-02-17T05:18:21.482 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67160.log.gz 2026-02-17T05:18:21.482 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52911.log 2026-02-17T05:18:21.482 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111056.log.gz 2026-02-17T05:18:21.482 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68434.log 2026-02-17T05:18:21.482 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53278.log.gz 2026-02-17T05:18:21.482 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.52911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70995.log 2026-02-17T05:18:21.482 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52911.log.gz 2026-02-17T05:18:21.482 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80114.log 2026-02-17T05:18:21.482 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68434.log.gz 2026-02-17T05:18:21.483 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85799.log 2026-02-17T05:18:21.483 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70995.log.gz 2026-02-17T05:18:21.483 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59840.log 2026-02-17T05:18:21.483 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80114.log.gz 2026-02-17T05:18:21.483 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85554.log 2026-02-17T05:18:21.483 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85799.log.gz 2026-02-17T05:18:21.483 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99001.log 2026-02-17T05:18:21.483 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59840.log.gz 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75753.log 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85554.log.gz 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76341.log 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99001.log.gz 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54302.log 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75753.log.gz 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97383.log 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76341.log.gz 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82662.log 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54302.log.gz 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97383.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.97383.log.gz --verbose 2026-02-17T05:18:21.484 INFO:teuthology.orchestra.run.trial100.stderr: -- /var/log/ceph/ceph-client.admin.55037.log 2026-02-17T05:18:21.485 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59987.log 2026-02-17T05:18:21.485 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82662.log.gz 2026-02-17T05:18:21.485 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109880.log 2026-02-17T05:18:21.485 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55037.log.gz 2026-02-17T05:18:21.485 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59987.log.gz 2026-02-17T05:18:21.485 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55184.log 2026-02-17T05:18:21.485 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109880.log.gz 2026-02-17T05:18:21.485 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88593.log 2026-02-17T05:18:21.485 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84525.log 2026-02-17T05:18:21.485 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55184.log.gz 2026-02-17T05:18:21.486 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109978.log 2026-02-17T05:18:21.486 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88593.log.gz 2026-02-17T05:18:21.486 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84525.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.84525.log.gz --verbose 2026-02-17T05:18:21.486 INFO:teuthology.orchestra.run.trial100.stderr: -- /var/log/ceph/ceph-client.admin.101745.log 2026-02-17T05:18:21.486 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98903.log 2026-02-17T05:18:21.486 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109978.log.gz 2026-02-17T05:18:21.486 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66912.log 2026-02-17T05:18:21.486 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101745.log.gz 2026-02-17T05:18:21.487 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48703.log 2026-02-17T05:18:21.487 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98903.log.gz 2026-02-17T05:18:21.487 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66912.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-17T05:18:21.487 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66912.log.gz 2026-02-17T05:18:21.487 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110664.log 2026-02-17T05:18:21.487 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48703.log.gz 2026-02-17T05:18:21.487 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80702.log 2026-02-17T05:18:21.487 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86583.log 2026-02-17T05:18:21.487 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110664.log.gz 2026-02-17T05:18:21.487 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114256.log 2026-02-17T05:18:21.487 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80702.log.gz 2026-02-17T05:18:21.488 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83152.log 2026-02-17T05:18:21.488 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86583.log.gz 2026-02-17T05:18:21.488 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113374.log 2026-02-17T05:18:21.488 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114256.log.gz 2026-02-17T05:18:21.488 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100716.log 2026-02-17T05:18:21.488 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83152.log.gz 2026-02-17T05:18:21.488 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66961.log 2026-02-17T05:18:21.488 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113374.log.gz 2026-02-17T05:18:21.488 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53277.log 2026-02-17T05:18:21.489 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100716.log.gz 2026-02-17T05:18:21.489 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.66961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89819.log 2026-02-17T05:18:21.489 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66961.log.gz 2026-02-17T05:18:21.489 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74577.log 2026-02-17T05:18:21.489 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53277.log.gz 2026-02-17T05:18:21.489 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87907.log 2026-02-17T05:18:21.489 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89819.log.gz 2026-02-17T05:18:21.489 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61016.log 2026-02-17T05:18:21.489 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74577.log.gz 2026-02-17T05:18:21.489 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91697.log 2026-02-17T05:18:21.489 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87907.log.gz 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51328.log 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61016.log.gz 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103607.log 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91697.log.gz 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52636.log 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51328.log.gz 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83447.log 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103607.log.gz 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.52636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73891.log 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52636.log.gz 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53146.log 2026-02-17T05:18:21.490 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83447.log.gz 2026-02-17T05:18:21.491 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83496.log 2026-02-17T05:18:21.491 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73891.log.gz 2026-02-17T05:18:21.491 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55625.log 2026-02-17T05:18:21.491 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53146.log.gz 2026-02-17T05:18:21.491 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63383.log 2026-02-17T05:18:21.491 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83496.log.gz 2026-02-17T05:18:21.491 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86485.log 2026-02-17T05:18:21.491 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55625.log.gz 2026-02-17T05:18:21.491 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100863.log 2026-02-17T05:18:21.491 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63383.log.gz 2026-02-17T05:18:21.492 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106008.log 2026-02-17T05:18:21.492 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86485.log.gz 2026-02-17T05:18:21.492 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111154.log 2026-02-17T05:18:21.492 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100863.log.gz 2026-02-17T05:18:21.492 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88838.log 2026-02-17T05:18:21.492 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106008.log.gz 2026-02-17T05:18:21.492 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52734.log 2026-02-17T05:18:21.492 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111154.log.gz 2026-02-17T05:18:21.492 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68679.log 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88838.log.gz 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.52734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105126.log 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52734.log.gz 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90459.log 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68679.log.gz 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.105126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75214.log 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105126.log.gz 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85701.log 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90459.log.gz 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52685.log 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75214.log.gz 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82711.log 2026-02-17T05:18:21.493 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85701.log.gz 2026-02-17T05:18:21.494 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.52685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62893.log 2026-02-17T05:18:21.494 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52685.log.gz 2026-02-17T05:18:21.494 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79036.log 2026-02-17T05:18:21.494 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82711.log.gz 2026-02-17T05:18:21.494 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91795.log 2026-02-17T05:18:21.494 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62893.log.gz 2026-02-17T05:18:21.494 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72568.log 2026-02-17T05:18:21.495 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79036.log.gz 2026-02-17T05:18:21.495 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64364.log 2026-02-17T05:18:21.495 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91795.log.gz 2026-02-17T05:18:21.495 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74626.log 2026-02-17T05:18:21.495 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72568.log.gz 2026-02-17T05:18:21.495 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69071.log 2026-02-17T05:18:21.495 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64364.log.gz 2026-02-17T05:18:21.495 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110419.log 2026-02-17T05:18:21.495 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74626.log.gz 2026-02-17T05:18:21.495 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88005.log 2026-02-17T05:18:21.495 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69071.log.gz 2026-02-17T05:18:21.496 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53812.log 2026-02-17T05:18:21.496 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110419.log.gz 2026-02-17T05:18:21.496 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87515.log 2026-02-17T05:18:21.496 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88005.log.gz 2026-02-17T05:18:21.496 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.53812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81241.log 2026-02-17T05:18:21.496 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53812.log.gz 2026-02-17T05:18:21.496 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85505.log 2026-02-17T05:18:21.496 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87515.log.gz 2026-02-17T05:18:21.496 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106547.log 2026-02-17T05:18:21.496 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81241.log.gz 2026-02-17T05:18:21.497 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91586.log 2026-02-17T05:18:21.497 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85505.log.gz 2026-02-17T05:18:21.497 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76733.log 2026-02-17T05:18:21.497 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106547.log.gz 2026-02-17T05:18:21.497 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94051.log 2026-02-17T05:18:21.497 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91586.log.gz 2026-02-17T05:18:21.497 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106253.log 2026-02-17T05:18:21.497 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76733.log.gz 2026-02-17T05:18:21.497 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69904.log 2026-02-17T05:18:21.497 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94051.log.gz 2026-02-17T05:18:21.497 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72027.log 2026-02-17T05:18:21.498 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106253.log.gz 2026-02-17T05:18:21.498 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78840.log 2026-02-17T05:18:21.498 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69904.log.gz 2026-02-17T05:18:21.498 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100912.log 2026-02-17T05:18:21.498 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72027.log.gz 2026-02-17T05:18:21.498 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65344.log 2026-02-17T05:18:21.498 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78840.log.gz 2026-02-17T05:18:21.498 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102382.log 2026-02-17T05:18:21.498 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100912.log.gz 2026-02-17T05:18:21.499 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61947.log 2026-02-17T05:18:21.499 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65344.log.gz 2026-02-17T05:18:21.499 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96158.log 2026-02-17T05:18:21.499 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102382.log.gz 2026-02-17T05:18:21.499 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112931.log 2026-02-17T05:18:21.499 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61947.log.gz 2026-02-17T05:18:21.499 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90802.log 2026-02-17T05:18:21.499 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96158.log.gz 2026-02-17T05:18:21.499 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89231.log 2026-02-17T05:18:21.500 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112931.log.gz 2026-02-17T05:18:21.500 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108655.log 2026-02-17T05:18:21.500 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90802.log.gz 2026-02-17T05:18:21.500 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86338.log 2026-02-17T05:18:21.500 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89231.log.gz 2026-02-17T05:18:21.500 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109537.log 2026-02-17T05:18:21.500 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108655.log.gz 2026-02-17T05:18:21.500 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86338.log.gz 2026-02-17T05:18:21.500 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56311.log 2026-02-17T05:18:21.500 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109537.log.gz 2026-02-17T05:18:21.500 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72274.log 2026-02-17T05:18:21.501 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60036.log 2026-02-17T05:18:21.501 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56311.log.gz 2026-02-17T05:18:21.501 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.72274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72274.log.gzgzip 2026-02-17T05:18:21.501 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.109586.log 2026-02-17T05:18:21.501 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88299.log 2026-02-17T05:18:21.501 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60036.log.gz 2026-02-17T05:18:21.501 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99491.log 2026-02-17T05:18:21.501 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109586.log.gz 2026-02-17T05:18:21.501 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99295.log 2026-02-17T05:18:21.501 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88299.log.gz 2026-02-17T05:18:21.501 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99491.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.99491.log.gz 2026-02-17T05:18:21.502 INFO:teuthology.orchestra.run.trial100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49978.log 2026-02-17T05:18:21.502 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98756.log 2026-02-17T05:18:21.502 INFO:teuthology.orchestra.run.trial100.stderr: 31.4% -- replaced with /var/log/ceph/ceph-client.admin.99295.log.gz 2026-02-17T05:18:21.502 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96550.log 2026-02-17T05:18:21.502 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49978.log.gz 2026-02-17T05:18:21.502 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79281.log 2026-02-17T05:18:21.502 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98756.log.gz 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87956.log 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96550.log.gz 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79134.log 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79281.log.gz 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84966.log 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87956.log.gz 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61751.log 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79134.log.gz 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87269.log 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84966.log.gz 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69806.log 2026-02-17T05:18:21.503 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61751.log.gz 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74822.log 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87269.log.gz 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44539.log 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69806.log.gz 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64118.log 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74822.log.gz 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.44539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75949.log 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44539.log.gz 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85897.log 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64118.log.gz 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70247.log 2026-02-17T05:18:21.504 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75949.log.gz 2026-02-17T05:18:21.505 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89476.log 2026-02-17T05:18:21.505 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85897.log.gz 2026-02-17T05:18:21.505 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114354.log 2026-02-17T05:18:21.505 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70247.log.gz 2026-02-17T05:18:21.505 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.89476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57781.log 2026-02-17T05:18:21.505 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89476.log.gz 2026-02-17T05:18:21.505 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.114354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102186.log 2026-02-17T05:18:21.505 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114354.log.gz 2026-02-17T05:18:21.505 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106449.log 2026-02-17T05:18:21.505 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57781.log.gz 2026-02-17T05:18:21.506 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102186.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-17T05:18:21.506 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102186.log.gz 2026-02-17T05:18:21.506 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90165.log 2026-02-17T05:18:21.506 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106449.log.gz 2026-02-17T05:18:21.506 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94688.log 2026-02-17T05:18:21.506 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65001.log 2026-02-17T05:18:21.506 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90165.log.gz 2026-02-17T05:18:21.506 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67846.log 2026-02-17T05:18:21.506 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94688.log.gz 2026-02-17T05:18:21.506 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63432.log 2026-02-17T05:18:21.506 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65001.log.gz 2026-02-17T05:18:21.507 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61310.log 2026-02-17T05:18:21.507 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67846.log.gz 2026-02-17T05:18:21.507 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50586.log 2026-02-17T05:18:21.507 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63432.log.gz 2026-02-17T05:18:21.507 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61310.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61310.log.gz -5 2026-02-17T05:18:21.507 INFO:teuthology.orchestra.run.trial100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.77223.log 2026-02-17T05:18:21.507 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98854.log 2026-02-17T05:18:21.507 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50586.log.gz 2026-02-17T05:18:21.507 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81339.log 2026-02-17T05:18:21.507 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77223.log.gz 2026-02-17T05:18:21.508 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81682.log 2026-02-17T05:18:21.508 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98854.log.gz 2026-02-17T05:18:21.508 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63726.log 2026-02-17T05:18:21.508 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81339.log.gz 2026-02-17T05:18:21.508 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.81682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81682.log.gz 2026-02-17T05:18:21.508 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103950.log 2026-02-17T05:18:21.508 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87466.log 2026-02-17T05:18:21.508 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63726.log.gz 2026-02-17T05:18:21.509 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67209.log 2026-02-17T05:18:21.509 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103950.log.gz 2026-02-17T05:18:21.509 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60232.log 2026-02-17T05:18:21.509 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87466.log.gz 2026-02-17T05:18:21.509 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.67209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100961.log 2026-02-17T05:18:21.509 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67209.log.gz 2026-02-17T05:18:21.509 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91145.log 2026-02-17T05:18:21.509 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60232.log.gz 2026-02-17T05:18:21.509 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65246.log 2026-02-17T05:18:21.509 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100961.log.gz 2026-02-17T05:18:21.510 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56262.log 2026-02-17T05:18:21.510 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91145.log.gz 2026-02-17T05:18:21.510 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65687.log 2026-02-17T05:18:21.510 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65246.log.gz 2026-02-17T05:18:21.510 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61114.log 2026-02-17T05:18:21.510 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56262.log.gz 2026-02-17T05:18:21.510 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65687.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.65687.log.gz -- 2026-02-17T05:18:21.510 INFO:teuthology.orchestra.run.trial100.stderr: /var/log/ceph/ceph-client.admin.88985.log 2026-02-17T05:18:21.510 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68189.log 2026-02-17T05:18:21.510 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61114.log.gz 2026-02-17T05:18:21.511 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95472.log 2026-02-17T05:18:21.511 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88985.log.gz 2026-02-17T05:18:21.511 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49309.log 2026-02-17T05:18:21.511 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68189.log.gz 2026-02-17T05:18:21.511 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.95472.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95472.log.gz -5 2026-02-17T05:18:21.511 INFO:teuthology.orchestra.run.trial100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.61898.log 2026-02-17T05:18:21.511 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112183.log 2026-02-17T05:18:21.511 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49309.log.gz 2026-02-17T05:18:21.511 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.61898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55919.log 2026-02-17T05:18:21.511 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61898.log.gz 2026-02-17T05:18:21.512 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60281.log 2026-02-17T05:18:21.512 INFO:teuthology.orchestra.run.trial100.stderr: 32.3% -- replaced with /var/log/ceph/ceph-client.admin.112183.log.gz 2026-02-17T05:18:21.512 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108361.log 2026-02-17T05:18:21.512 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55919.log.gz 2026-02-17T05:18:21.512 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76194.log 2026-02-17T05:18:21.512 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60281.log.gz 2026-02-17T05:18:21.512 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56458.log 2026-02-17T05:18:21.512 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108361.log.gz 2026-02-17T05:18:21.512 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76194.log.gz 2026-02-17T05:18:21.513 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74087.log 2026-02-17T05:18:21.513 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70002.log 2026-02-17T05:18:21.513 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56458.log.gz 2026-02-17T05:18:21.513 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.74087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110566.log 2026-02-17T05:18:21.513 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74087.log.gz 2026-02-17T05:18:21.513 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54498.log 2026-02-17T05:18:21.513 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70002.log.gz 2026-02-17T05:18:21.513 INFO:teuthology.orchestra.run.trial100.stderr: 95.6% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-17T05:18:21.513 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56752.log 2026-02-17T05:18:21.513 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110566.log.gz 2026-02-17T05:18:21.514 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69218.log 2026-02-17T05:18:21.514 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54498.log.gz 2026-02-17T05:18:21.514 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.56752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78987.log 2026-02-17T05:18:21.514 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56752.log.gz 2026-02-17T05:18:21.514 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.69218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69218.log.gz 2026-02-17T05:18:21.514 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63971.log 2026-02-17T05:18:21.514 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63873.log 2026-02-17T05:18:21.514 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78987.log.gz 2026-02-17T05:18:21.514 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76047.log 2026-02-17T05:18:21.514 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63971.log.gz 2026-02-17T05:18:21.515 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112232.log 2026-02-17T05:18:21.515 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63873.log.gz 2026-02-17T05:18:21.515 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83398.log 2026-02-17T05:18:21.515 INFO:teuthology.orchestra.run.trial100.stderr: 32.0% -- replaced with /var/log/ceph/ceph-client.admin.76047.log.gz 2026-02-17T05:18:21.515 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.112232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82417.log 2026-02-17T05:18:21.515 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112232.log.gz 2026-02-17T05:18:21.515 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102333.log 2026-02-17T05:18:21.515 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83398.log.gz 2026-02-17T05:18:21.515 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110272.log 2026-02-17T05:18:21.515 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82417.log.gz 2026-02-17T05:18:21.516 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92285.log 2026-02-17T05:18:21.516 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102333.log.gz 2026-02-17T05:18:21.516 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104342.log 2026-02-17T05:18:21.516 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110272.log.gz 2026-02-17T05:18:21.516 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.92285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57487.log 2026-02-17T05:18:21.516 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92285.log.gz 2026-02-17T05:18:21.516 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104342.log.gz 2026-02-17T05:18:21.516 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94541.log 2026-02-17T05:18:21.516 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94982.log 2026-02-17T05:18:21.517 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57487.log.gz 2026-02-17T05:18:21.517 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104195.log 2026-02-17T05:18:21.517 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94541.log.gz 2026-02-17T05:18:21.517 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94982.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.59055.log -- replaced with /var/log/ceph/ceph-client.admin.94982.log.gz 2026-02-17T05:18:21.517 INFO:teuthology.orchestra.run.trial100.stderr: 2026-02-17T05:18:21.517 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.104195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110468.log 2026-02-17T05:18:21.517 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104195.log.gz 2026-02-17T05:18:21.517 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.59055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85260.log 2026-02-17T05:18:21.517 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59055.log.gz 2026-02-17T05:18:21.517 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76243.log 2026-02-17T05:18:21.517 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110468.log.gz 2026-02-17T05:18:21.518 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79428.log 2026-02-17T05:18:21.518 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.85260.log.gz 2026-02-17T05:18:21.518 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.76243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115334.log 2026-02-17T05:18:21.518 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76243.log.gz 2026-02-17T05:18:21.518 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.79428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50658.log 2026-02-17T05:18:21.518 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79428.log.gz 2026-02-17T05:18:21.518 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.115334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70554.log 2026-02-17T05:18:21.518 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115334.log.gz 2026-02-17T05:18:21.518 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88201.log 2026-02-17T05:18:21.518 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90753.log 2026-02-17T05:18:21.519 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70554.log.gz 2026-02-17T05:18:21.519 INFO:teuthology.orchestra.run.trial100.stderr: 96.1% -- replaced with /var/log/ceph/ceph-client.admin.50658.log.gz 2026-02-17T05:18:21.519 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.88201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100177.log 2026-02-17T05:18:21.519 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88201.log.gz 2026-02-17T05:18:21.519 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99344.log 2026-02-17T05:18:21.519 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90753.log.gz 2026-02-17T05:18:21.519 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.100177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73695.log 2026-02-17T05:18:21.519 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100177.log.gz 2026-02-17T05:18:21.519 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87858.log 2026-02-17T05:18:21.519 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99344.log.gz 2026-02-17T05:18:21.520 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.73695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60575.log 2026-02-17T05:18:21.520 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73695.log.gz 2026-02-17T05:18:21.520 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.87858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111840.log 2026-02-17T05:18:21.520 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87858.log.gz 2026-02-17T05:18:21.520 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48574.log 2026-02-17T05:18:21.520 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60575.log.gz 2026-02-17T05:18:21.520 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75165.log 2026-02-17T05:18:21.520 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111840.log.gz 2026-02-17T05:18:21.520 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.48574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96011.log 2026-02-17T05:18:21.520 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48574.log.gz 2026-02-17T05:18:21.521 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.75165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77174.log 2026-02-17T05:18:21.521 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75165.log.gz 2026-02-17T05:18:21.521 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108165.log 2026-02-17T05:18:21.521 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96011.log.gz 2026-02-17T05:18:21.521 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77174.log.gz 2026-02-17T05:18:21.521 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68630.log 2026-02-17T05:18:21.521 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77419.log 2026-02-17T05:18:21.521 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108165.log.gz 2026-02-17T05:18:21.521 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.68630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110958.log 2026-02-17T05:18:21.521 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68630.log.gz 2026-02-17T05:18:21.521 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107919.log 2026-02-17T05:18:21.522 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77419.log.gz 2026-02-17T05:18:21.522 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.110958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85211.log 2026-02-17T05:18:21.522 INFO:teuthology.orchestra.run.trial100.stderr: 29.5% -- replaced with /var/log/ceph/ceph-client.admin.110958.log.gz 2026-02-17T05:18:21.522 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96207.log 2026-02-17T05:18:21.522 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107919.log.gz 2026-02-17T05:18:21.522 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.85211.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.85211.log.gz /var/log/ceph/ceph-client.admin.55478.log 2026-02-17T05:18:21.522 INFO:teuthology.orchestra.run.trial100.stderr: 2026-02-17T05:18:21.522 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103852.log 2026-02-17T05:18:21.523 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96207.log.gz 2026-02-17T05:18:21.523 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.55478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83937.log 2026-02-17T05:18:21.523 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55478.log.gz 2026-02-17T05:18:21.523 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103852.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.94835.log 2026-02-17T05:18:21.523 INFO:teuthology.orchestra.run.trial100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.103852.log.gz 2026-02-17T05:18:21.523 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.83937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111301.log 2026-02-17T05:18:21.523 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83937.log.gz 2026-02-17T05:18:21.523 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102578.log 2026-02-17T05:18:21.523 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94835.log.gz 2026-02-17T05:18:21.523 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.111301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103019.log 2026-02-17T05:18:21.523 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111301.log.gz 2026-02-17T05:18:21.524 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.102578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107429.log 2026-02-17T05:18:21.524 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102578.log.gz 2026-02-17T05:18:21.524 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84868.log 2026-02-17T05:18:21.524 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103019.log.gz 2026-02-17T05:18:21.524 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57928.log 2026-02-17T05:18:21.524 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107429.log.gz 2026-02-17T05:18:21.524 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103803.log 2026-02-17T05:18:21.524 INFO:teuthology.orchestra.run.trial100.stderr: 31.4% -- replaced with /var/log/ceph/ceph-client.admin.84868.log.gz 2026-02-17T05:18:21.524 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.57928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58173.log 2026-02-17T05:18:21.524 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57928.log.gz 2026-02-17T05:18:21.525 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54792.log 2026-02-17T05:18:21.525 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103803.log.gz 2026-02-17T05:18:21.525 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54694.log 2026-02-17T05:18:21.525 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58173.log.gz 2026-02-17T05:18:21.525 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113129.log 2026-02-17T05:18:21.525 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54792.log.gz 2026-02-17T05:18:21.525 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.54694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99589.log 2026-02-17T05:18:21.525 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54694.log.gz 2026-02-17T05:18:21.525 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.113129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77321.log 2026-02-17T05:18:21.525 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113129.log.gz 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.99589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91942.log 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99589.log.gz 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109488.log 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77321.log.gz 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.91942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51577.log 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91942.log.gz 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.109488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108214.log 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109488.log.gz 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.51577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93167.log 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51577.log.gz 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.108214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107723.log 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108214.log.gz 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62241.log 2026-02-17T05:18:21.526 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93167.log.gz 2026-02-17T05:18:21.527 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.107723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84133.log 2026-02-17T05:18:21.527 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107723.log.gz 2026-02-17T05:18:21.527 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62403.log 2026-02-17T05:18:21.527 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62241.log.gz 2026-02-17T05:18:21.527 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.84133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70701.log 2026-02-17T05:18:21.527 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84133.log.gz 2026-02-17T05:18:21.527 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.62403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77370.log 2026-02-17T05:18:21.527 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62403.log.gz 2026-02-17T05:18:21.528 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.70701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97727.log 2026-02-17T05:18:21.528 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70701.log.gz 2026-02-17T05:18:21.528 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.77370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78056.log 2026-02-17T05:18:21.528 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77370.log.gz 2026-02-17T05:18:21.528 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97138.log 2026-02-17T05:18:21.528 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97727.log.gz 2026-02-17T05:18:21.528 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.78056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106400.log 2026-02-17T05:18:21.528 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78056.log.gz 2026-02-17T05:18:21.528 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.97138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101059.log 2026-02-17T05:18:21.528 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97138.log.gz 2026-02-17T05:18:21.528 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96060.log 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106400.log.gz 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.101059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65981.log 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101059.log.gz 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.96060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60134.log 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr: 32.0% -- replaced with /var/log/ceph/ceph-client.admin.96060.log.gz 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93561.log 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65981.log.gz 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64952.log 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60134.log.gz 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.93561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94786.log 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93561.log.gz 2026-02-17T05:18:21.529 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.64952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98413.log 2026-02-17T05:18:21.530 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64952.log.gz 2026-02-17T05:18:21.530 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.94786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60526.log 2026-02-17T05:18:21.530 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94786.log.gz 2026-02-17T05:18:21.530 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.98413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90704.log 2026-02-17T05:18:21.530 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98413.log.gz 2026-02-17T05:18:21.530 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.60526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58124.log 2026-02-17T05:18:21.530 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60526.log.gz 2026-02-17T05:18:21.530 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.90704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86436.log 2026-02-17T05:18:21.530 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90704.log.gz 2026-02-17T05:18:21.531 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63628.log 2026-02-17T05:18:21.531 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58124.log.gz 2026-02-17T05:18:21.531 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.86436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80947.log 2026-02-17T05:18:21.531 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86436.log.gz 2026-02-17T05:18:21.531 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.63628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82858.log 2026-02-17T05:18:21.531 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63628.log.gz 2026-02-17T05:18:21.531 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106155.log 2026-02-17T05:18:21.531 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80947.log.gz 2026-02-17T05:18:21.531 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.82858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103754.log 2026-02-17T05:18:21.531 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82858.log.gz 2026-02-17T05:18:21.531 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.106155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49606.log 2026-02-17T05:18:21.532 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106155.log.gz 2026-02-17T05:18:21.532 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.103754.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103754.log.gz -5 2026-02-17T05:18:21.532 INFO:teuthology.orchestra.run.trial100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58516.log 2026-02-17T05:18:21.532 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.49606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50393.log 2026-02-17T05:18:21.532 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49606.log.gz 2026-02-17T05:18:21.532 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.58516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80849.log 2026-02-17T05:18:21.532 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58516.log.gz 2026-02-17T05:18:21.532 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.50393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65932.log 2026-02-17T05:18:21.532 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50393.log.gz 2026-02-17T05:18:21.532 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.80849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80849.log.gz 2026-02-17T05:18:21.532 INFO:teuthology.orchestra.run.trial100.stderr:/var/log/ceph/ceph-client.admin.65932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65932.log.gz 2026-02-17T05:18:21.633 INFO:teuthology.orchestra.run.trial100.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-17T05:18:21.744 INFO:teuthology.orchestra.run.trial100.stderr: 87.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2026-02-17T05:18:21.896 INFO:teuthology.orchestra.run.trial100.stderr: 87.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2026-02-17T05:18:21.963 INFO:teuthology.orchestra.run.trial100.stderr: 87.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-17T05:18:22.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2026-02-17T05:18:36.400 INFO:teuthology.orchestra.run.trial100.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-17T05:18:47.067 INFO:teuthology.orchestra.run.trial100.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-17T05:19:15.506 INFO:teuthology.orchestra.run.trial100.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-17T05:19:15.508 INFO:teuthology.orchestra.run.trial100.stderr: 2026-02-17T05:19:15.508 INFO:teuthology.orchestra.run.trial100.stderr:real 0m54.308s 2026-02-17T05:19:15.508 INFO:teuthology.orchestra.run.trial100.stderr:user 1m33.284s 2026-02-17T05:19:15.508 INFO:teuthology.orchestra.run.trial100.stderr:sys 0m4.299s 2026-02-17T05:19:15.508 INFO:tasks.ceph:Archiving logs... 2026-02-17T05:19:15.509 DEBUG:teuthology.misc:Transferring archived files from trial100:/var/log/ceph to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_03:53:03-rados-wip-bharath18-testing-2026-02-13-0856-distro-default-trial/50912/remote/trial100/log 2026-02-17T05:19:15.511 DEBUG:teuthology.orchestra.run.trial100:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-17T05:19:24.042 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_325850157d309f65985eea9ec9bb9f1d56726fee/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_325850157d309f65985eea9ec9bb9f1d56726fee/qa/tasks/ceph_manager.py", line 2947, in wait_for_clean assert time.time() - start < timeout, \ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: wait_for_clean: failed before timeout expired 2026-02-17T05:19:24.043 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-17T05:19:24.063 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_325850157d309f65985eea9ec9bb9f1d56726fee/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_325850157d309f65985eea9ec9bb9f1d56726fee/qa/tasks/ceph_manager.py", line 2947, in wait_for_clean assert time.time() - start < timeout, \ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: wait_for_clean: failed before timeout expired 2026-02-17T05:19:24.064 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-17T05:19:24.064 DEBUG:teuthology.orchestra.run.trial100:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-17T05:19:24.161 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-17T05:19:24.172 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-17T05:19:24.173 DEBUG:teuthology.orchestra.run.trial100:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-17T05:19:24.189 INFO:teuthology.orchestra.run.trial100.stderr:bash: line 1: ntpq: command not found 2026-02-17T05:19:24.219 INFO:teuthology.orchestra.run.trial100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-17T05:19:24.220 INFO:teuthology.orchestra.run.trial100.stdout:=============================================================================== 2026-02-17T05:19:24.220 INFO:teuthology.orchestra.run.trial100.stdout:^- host-131-239-5-43.custom> 2 7 377 125 +1689us[+1689us] +/- 28ms 2026-02-17T05:19:24.220 INFO:teuthology.orchestra.run.trial100.stdout:^- 23.186.168.132 2 7 373 189 +416us[ +416us] +/- 56ms 2026-02-17T05:19:24.220 INFO:teuthology.orchestra.run.trial100.stdout:^* 149.28.61.105.vultruserc> 3 7 377 255 -23us[ -28us] +/- 3641us 2026-02-17T05:19:24.220 INFO:teuthology.orchestra.run.trial100.stdout:^- ntp11.kernfusion.at 2 6 377 58 -2365us[-2365us] +/- 75ms 2026-02-17T05:19:24.220 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-17T05:19:24.232 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-17T05:19:24.232 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-17T05:19:24.246 DEBUG:teuthology.orchestra.run.trial100:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-17T05:19:24.333 DEBUG:teuthology.orchestra.run.trial100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-17T05:19:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-17T05:19:24.361 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-17T05:19:24.376 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-17T05:19:24.391 INFO:teuthology.task.internal:Duration was 1791.973978 seconds 2026-02-17T05:19:24.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-17T05:19:24.408 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-17T05:19:24.408 DEBUG:teuthology.orchestra.run.trial100:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-17T05:19:24.445 INFO:teuthology.orchestra.run.trial100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-17T05:19:24.654 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-17T05:19:24.654 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial100.front.sepia.ceph.com 2026-02-17T05:19:24.654 DEBUG:teuthology.orchestra.run.trial100:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-17T05:19:24.712 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-17T05:19:24.713 DEBUG:teuthology.orchestra.run.trial100:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-17T05:19:24.835 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-17T05:19:24.835 DEBUG:teuthology.orchestra.run.trial100:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-17T05:19:24.860 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-17T05:19:24.860 INFO:teuthology.orchestra.run.trial100.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-17T05:19:24.860 INFO:teuthology.orchestra.run.trial100.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-17T05:19:24.860 INFO:teuthology.orchestra.run.trial100.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-17T05:19:24.860 INFO:teuthology.orchestra.run.trial100.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-17T05:19:24.872 INFO:teuthology.orchestra.run.trial100.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 87.2% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-17T05:19:24.874 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-17T05:19:24.892 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-17T05:19:24.892 DEBUG:teuthology.orchestra.run.trial100:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-17T05:19:24.941 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-17T05:19:24.956 DEBUG:teuthology.orchestra.run.trial100:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-17T05:19:25.005 INFO:teuthology.orchestra.run.trial100.stdout:kernel.core_pattern = core 2026-02-17T05:19:25.022 DEBUG:teuthology.orchestra.run.trial100:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-17T05:19:25.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-17T05:19:25.078 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-17T05:19:25.093 INFO:teuthology.task.internal:Transferring archived files... 2026-02-17T05:19:25.094 DEBUG:teuthology.misc:Transferring archived files from trial100:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_03:53:03-rados-wip-bharath18-testing-2026-02-13-0856-distro-default-trial/50912/remote/trial100 2026-02-17T05:19:25.094 DEBUG:teuthology.orchestra.run.trial100:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-17T05:19:25.150 INFO:teuthology.task.internal:Removing archive directory... 2026-02-17T05:19:25.150 DEBUG:teuthology.orchestra.run.trial100:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-17T05:19:25.203 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-17T05:19:25.221 INFO:teuthology.task.internal:Not uploading archives. 2026-02-17T05:19:25.222 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-17T05:19:25.238 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-17T05:19:25.239 DEBUG:teuthology.orchestra.run.trial100:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-17T05:19:25.258 INFO:teuthology.orchestra.run.trial100.stdout: 791885 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 17 05:19 /home/ubuntu/cephtest 2026-02-17T05:19:25.260 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-17T05:19:25.277 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-17T05:19:25.294 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: AssertionError: wait_for_clean: failed before timeout expired 2026-02-17T05:19:25.301 INFO:teuthology.run:Summary data: description: rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/many msgr/async-v2only objectstore/{bluestore/{alloc$/{stupid} base mem$/{normal-1} onode-segment$/{1M} write$/{v2/{compr$/{no$/{no}} v2}}}} rados supported-random-distro$/{centos_latest}} duration: 1791.9739780426025 failure_reason: '' flavor: default owner: scheduled_skanta@soko04.front.sepia.ceph.com sentry_event: null status: fail success: false 2026-02-17T05:19:25.301 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-17T05:19:25.414 INFO:teuthology.run:FAIL